The Intriguing Effects of Focal Loss on the Calibration of Deep Neural Networks
Authors: Jishnu Mukhoti
Viveka Kulharia
Amartya Sanyal
Stuart Golodetz
Philip Torr
Puneet Dokania
Venue:  (2020)

Abstract: Miscalibration -- a mismatch between a model's confidence and its correctness -- of Deep Neural Networks (DNNs) makes their predictions hard for downstream components to trust. Ideally, we want networks to be accurate, calibrated and confident. Temperature scaling, the most popular calibration approach, will calibrate a DNN without affecting its accuracy, but it will also make its correct predictions under-confident. In this paper, we show that replacing the widely used cross-entropy loss with focal loss allows us to learn models that are already very well calibrated. When combined with temperature scaling, focal loss, whilst preserving accuracy and yielding state-of-the-art calibrated models, also preserves the confidence of the model's correct predictions, which is extremely desirable for downstream tasks. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to theoretically justify the empirically excellent performance of focal loss. We perform extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (SST, 20 Newsgroup) datasets, and with a wide variety of different network architectures, and show that our approach achieves state-of-the-art accuracy and calibration in almost all cases.

Under review as a conference paper at ICLR 2020
THE INTRIGUING EFFECTS OF FOCAL LOss ON THE CALIBRATION OF DEEP NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Miscalibration mismatch between a model's confidence and its correctness of Deep Neural Networks (DNNs) makes their predictions hard for downstream com- ponents to trust. Ideally, we want networks to be accurate, calibrated and confident. Temperature scaling, the most popular calibration approach, will calibrate DNN without affecting its accuracy, but it will also make its correct predictions under- confident. In this paper, we show that replacing the widely used cross-entropy loss with focal loss allows us to learn models that are already very well calibrated. When combined with temperature scaling, focal loss, whilst preserving accuracy and yielding state-of-the-an calibrated models, also preserves the confidence of the model's correct predictions, which is extremely desirable for downstream tasks. We provide thorough analysis of the factors causing miscalibration, and use the insights we glean from this to theoretically justify the empirically excellent perfor- mance of focal loss. We perform extensive experiments on a variety of computer vision (CIFAR-10/10) variety of different network architectures, and show that our approach achieves state-of-the-an accuracy and calibration in almost all cases.
and NLP (SST, 20 Newsgroup) datasets, and with a wide
INTRODUCTION
Deep neural networks have dominated computer vision and machine learning in recent years, and this has led their widespread deployment in real-world systems (Cao et al., 2018; Chen et al., 2018; Kamilaris and Prenafeta-Boldi, 2018; Ker et al., 2018; Wang et al., 2018). State-of-the-art networks achieve high levels of accuracy for many tasks. However, many current multi-class classification networks in particular are poorly calibrated, in the sense that the probability values that they associate with the class labels they predict for different test samples overestimate the likelihoods of those class labels being correct in the real world. This is a major problem, since if networks are routinely overconfident, then downstream components cannot trust their predictions. The underlying cause is potttess to be that these networks' high capacity leaves them vulnerable to overfitting on the negative log-likelihnn likelihood (NLL) loss they conventionally use during training (Guo* et al., 2017).
Given the importance of this problem, numerous suggestions for how to address it have been proposed. Much work has been inspired by early approaches from the pre-deep learning era such as Platt scaling (Platt, 1999), histogram binning (Zadrozny and Elkan, 2001), isotonic regression (Zadrozny and Elkan, 2002), and Bayesian binning and averaging (Naeini et al., 2015; Naeini and Cooper, 2016). As deep learning has become more dominant, works have begun directly target the calibration of deep networks. For example, Guo et al. (Guo* et al., 2017) have popularised a modern variant of Platt scaling known as temperature scaling, which works by dividing network's 's logits by scalar T > O (learnt on a validation subset) prior to performing softmax. Temperature scaling has the desirable property that can improve the calibration of a network without in any way affecting its accuracy. Mozafari et al. (Mozafari et al., 2018) noted the downsides of using cross-entropy loss with temperature scaling, and proposed an alternative loss called Attended-NL. that helps temperature scaling achieve better calibration. More recently, Shrikumar and Kundaje (Shrikumar and Kundaje, 2019) have proposed an extension to temperature scaling that adds class-specific bias parameters to help eliminate systematic bias when performing domain adaptation. Separately, Hendrycks et al. (Hendrycks et al., 2019) have studied the effects of pre-training (vs. training from scratch) on model robustness and uncertainty: they make the interesting observation that because long periods of training can cause a network to become miscalibrated, tuning a pre-trained network, which facilitates faster convergence, can seemingly lead to more calibrated model. Notably, since their approach

---

Under review as a conference paper at ICLR 2020
complements temperature scaling, the two techniques can also be used together to achieve even better calibration overall.
Whilst temperature scaling's simplicity and effectiveness have made it a popular and state-of-the-an art network calibration technique, does have downsides. For example, whilst it scales the logits to reduce the network's confidence in incorrect predictions, this also slightly reduces the network's confidence in predictions that were actually correct. By contrast, Kumar et al. (Kumar et al., 2018) eschew temperature scaling altogether in favour of minimising a differentiable proxy for calibration error at training time, called Maximum Mean Calibration Error (MMCE). However, they also use temperature scaling as a prosessoosssing step to obtain better results than cross-entropy followed by temperature scaling (Guo* et al., 2017).
In this paper, we propose a technique for improving network calibration that works by replacing the cross-entropy loss conventionally used when training multi-class classification networks with the focal loss proposed by Lin et al. (Lin et al., 2017) for dense object detection. Since focal loss, as shown in $4, is dependent on hyperparameter, 7, which needs to be cross-validated, we also provide a theoretically justified way to choose y automatically for each sample and show it to outperform all the baseline models.
Informally, the intuition behind using focal loss to direct the network's s attention during training towards samples for which it is currently predicting low probability value for the correct class, since trying to reduce the NLL on samples for which is currently predicting a high probability value for the correct class is liable to lead to NLL overfitting, and thereby miscalibration (Guo" et al., 2017). More formally, we show in $4 that focal loss can be seen as implicitly regularising the weights of the network by causing the gradient norm be lower than it would have been with cross-entropy loss as training proceeds, which we would theoretically expect to reduce overfitting and improve the calibration of the network. In $5, we perform extensive experiments on a variety of computer vision (CIFAR-10/ 10/100 and NLP (20 Newsgroups/SSTT datasets, and with a wide variety of different network architectures (e.g. ResNet-1 10/50, Wide-ResNet DenseNet), show that this is indeed the case.
Our experiments show that in almost all cases, DNNs trained with focal loss are more calibrated than those trained with cross coos-entropy loss, MMCE, and Brier loss Brier (1950). Moreover, since our approach, like that of (Hendrycks et al., 2019), is complementary to the temperature scaling, significant improvements in calibration over temperature scaling alone, and state-of-the-ard results, can be achieved by training with focal loss and then performing temperature scaling.
PROBLEM SETUP
Let D = ((x,, y denote a dataset consisting of samples from a joint distribution D(X, ). where for each sample i, x; E X' is the input and y; y = (1, ..., K} } is the ground-truth class label. Let p.,y = fo xx) be the probability that a neural network f with model parameters 0 predicts for class y on given input x, The class that ff predicts for x; computed as ÿ, = argmax Pr.y- and the predicted confidence as P. = max Pt.y- Note that the confidence, by definition, does not depend on the ground-truth label. The network is said to be perfectly culibrated when, for each sample (x,, ) E D, the confidence D; is equal to the model accuracy P(ù: yi), i.e. the probability that the predicted class is correct. For instance, of all the samples that a perfectly calibrated neural network classifies with confidence of 0.8, 80% should be correctly predicted.
One of the most popular metrics used to measure model calibration is the expected calibration error (ECE) (Nacini et al., 2015). defined as the expected absolute difference between the model's confidence and its accuracy. i.e. B(x [P = y4) - pal]. Since calculating P(js = yt) is infeasible, the ECE cannot be directly computed. A workaround is to divide the interval [0, 1] into M equispaced bins, where the 7th bin the interval (²-1 1. Let B, denote the set of samples with confidence scores belonging to the tth bin. Then the accuracy A; of this bin is computed as A, B. ΣYEB, 1 (Ø) y;). where is the indicator function, and Ú, and yj are respectively the predicted and ground-trute labels for the jth sample. Similarly, the confidence C; of the 7th bin is computed as C. [., ExEB, P.. i.e. C2 the average confidence of all samples in the bin. The ECE can be approximated as a weighted average of the absolute difference between the accuracy and confidence of each bin:
ECE DM M B 14 -al-
2

---

Under review as a conference paper at ICLR 2020
A similar metric, the maximumm calibration error (MCE) (Nacini al., 2015), is defined as the maximum absolute difference between the confidence and accuracy of each bin:
MCE (l... | Cal.
(2)
A common way of visually exploring the calibration of model is to use reliability plot (Niculescu- Mizil and Caruana, 2005), which plots the accuracies of the various confidence bins as a bar chart (see Figure 2). Reliability plots only give a quick visual indication of model's MCE (but not ECE, since the numbers of samples in the different bins are not shown), but also capture whether or not model is under-confident or over-confiennt in general. For a perfectly calibrated model, the accuracy for each bin will match the confidence. and hence all of the bars will lie on a diagonal. By contrast, if most of the bars lie above the diagonal, meaning that the model is more accurate than expects, then it is said to be under-confident, and if most of the bars lie below the diagonal, then it is said to be over-confident.
We also consider another metric which we call AdaECE (Adaptive ECE) where the bin sizes are decided such that each bin contains the same number of samples, unlike ECE (1) where every bin size is fixed and might contain very different number of samples (for example, Figure 2). However, AdaECE allows better approximation of ECE as it ensures that every approximate bin has the same sample density.
AdaECE
MM
i=
M
B.J
|A, - Cil where Vi + [ - |Byl-
(3)
3
WHAT CAUSES MISCALIBRATION?
We now discuss why high-capacity neural networks, despite achieving low classification errors on well-known datasets, tend to be miscalibrated. A key empirical observation made by (Guo* et al., 2017) was that poor calibration of such networks appears be linked overfitting on the negative log-like ligliikl (NLL) during training. In this section, we further inspect this observation to provide new insights, and discuss the main factors that play a role in causing the above-mentioned NI.L overfitting.
For the analysis in this section, we train ResNet-50 network on CIFAR- 10 using a training infras- tructure Py CyTA which is known to produce ofate-of-the-art accuracy. We train it for 350 epochs using Stochastic Gradient Descent (SGD), with a momentum of 0.9, and a learning rate of 0. for the first 150 epochs, 0.01 for the next 100 epochs and 0.001 for the last 100 epochs. We use mini-batch size of 128. Following standard practice, we minimise cross-entiee cross-entropy loss, which, for the log Pi.gs where Pi,y, is the probability assigned by the
tth training sample, is computed as L. network to the correct class for the 5th sample.
is interesting to note that NLL (a.k.a. cross cross-entropy) loss is a differentiable proxy for the actual metric that we aim to minimise by training model, namely the classification error. However, there are certain differences between minimising the NLL and minimising the classification error. For instance, the NIL is minimised when for every training sample i, Pi.ys 1, whereas the classification error is minimised if for every sample i, Pt.g. > Pr, for all # yi. This indicates that even when the classification error is zero. the NLL can be positive. and the optimisation algorithm could in that case try to reduce it to zero by further increasing the value of Pt.y. for each sample.
To study how miscalibration occurs during training, we plot the average NLL for the train and test sets at each training epoch in Figures ((a) and 1(b). We also plot the average NLL and the entropy of the softmax distribution produced by the network for the correctly and incorrectly classified samples. In Figure (c), we plot the classification emror on the train and test sets, along with the test set ECE.
Curse of misclassified samples: Figures ((a) and 1(b) show that although the average train NLL (for both correctly and incorrectly classified training samples) broadly decreases throughout training. after the th epoch (where the learning rate drops by a factor of 10), there is a marked rise in the average test NLL, indicating that the network starts to overfit on average NLL. However, the increase in average test NLL is caused only by the incorrectly classified samples, as the average NLL for the correctly classified samples continues to decrease after epoch 150, the test set ECE rises, indicating that the network is becoming miscalibrated. This corroborates the observation in (Guo* et al., 2017) that miscalibration and NLL overfitting are linked.
after the 150th epoch. We also observe that
3

---

Under review as a conference paper at ICLR 2020
Entropy
Train NLL correct
1.6
Test NILL correct
0..6
Train classification error classification
1.2
Trai NLL incorrect
Test NLLL incorrect
0.3
Train NLL
1.2
Test
Test ECE
1.0
0.8
Train Entropy Incerrect
Irem Entropy
1.0
0.8
Test Entropy Incorrect
Text Entrapy cornect
Train ertor Text
0.6
0.6
0.4
0.4
0.2
0.0
0.1
0.0
0.0
0.0
1.00 200 250 350
200 250 Training spoch
150
200 250 300 Training epoch
Training epoch
(a)
(b)
(c)
Figure 1: Metries to capture how model calibration changes over training epochs in ResNet-50 network trained on CIFAR-10.
Espected Achuul
Exseced
Espectedd Actiol
Sopected
Acheel
52
64
044
Confidence
06
5.8
62
04
5.5 Confidece
5.5
Peroriage sarples
Recortage
Contonice
03
10
02
Conticeni
3.5
08
1.0
Conficncce 33 epocls
2.1 6.5
100
300
32
Denfeence
64
08
10
100
200cs
390 cpocks
Figure 2: Top row: reliability plots using 25 confidence bins. Bottom row: % of samples in each bin, computed over the training at epochs 100, 200, 300 and 350.
Peak at the wrong place: We further observe that the entropies of the softmax distributions for both the correctly and incorrectly classified test samples decrease throughout training (in other words, the distributions get peakier). This observation, coupled with the one we made above, indicates that for the wrongly classified test samples, the network gradually becomes more and more confident about its incorrect predictions. Finally, we notice that the training set classification error drops from around 20% to nearly 0% at epoch 150, and further drops to exactly 0% from epoch 250 onwards. The network thus has very low training set classification error throughout the period of miscalibration.
Favouring correctly classified samples: One potential explanation for the above observations is that after the 150th training epoch, the network classifies almost all of the training samples correctly, and hence, in order to further minimise the training NLL., it focuses on increasing the confidence of its correct predictions, rather than focusing the incorrectly classified samples. In order to experimentally validate this, we use two observations. Firstly, we divide the confidence range [o, 1] into 25 bins, and present reliability plots computed on the training at epochs 100, 200, 300 and 350 (see the top row of Figure 2). In Figure 2, we also show the percentage of samples in each confidence bin. It is quite clear from these plots that over time, the network gradually pushes all of the training samples towards the highest confidence bin. Furthermore, even though the network has achieved 100% accuracy on the training set by epoch 300, still pushes some of the samples lying in lower confidence bins to the highest confidence bin by epoch 350. It thus keeps on increasing the confidence of correct predictions even after having reached perfect accuracy on the training set. Secondly, we observe from Figure I(c) that between training epochs 150 and 250, i.e. between the two points where there are drops in the learning rate, the training set classification error rises slightly, before dropping to exactly 0 after epoch 250. This further confirms hypothesis that in this period, the network focuses more on increasing the confidence of its correct predictions rather than increasing the classification accuracy on the training set.
4

---

Under review as a conference paper at ICLR 2020
0.40
Pocal (p-2.0
12
2,1
5.
5..25
Pocal (pm2.))
Food syel R
Foot sye2 E
3.8
Cyen
Cym
Cye
3.0
0.20
0.15
0.10
0.4
5o Lea
150
200 2000 300 2016
trakiigg
150
200 250 500
5o
150
203 250 100 200
50 103 150 200 200
320 X06
(a)
(b)
(c)
0.0
0.00
0.E79
houd (yma of
0.190
Tocal 1y-2 ot
01225
0.6
Tocal (ye3 0t
0 0,00
3 0.4
0.5
Tooal (y=2.0)
0073
0020
fooal 2,0)
6:22
Focal (1-3
192 230
300 350
Thm ning mo0ch
250
0000
Thaniig 1poch
350
Trening specy
(e)
(f)
(h)
Figure 3: Metrics to compare the calibrations of several ResNet-50 networks trained on CIFAR-10. 10, using either cross-entropy loss, or focal loss with y set to 1, 2,2 or 3.
Weight magnification: The above-mentions can happen if the network increases its weights to increase the magnitudes of the logits. We explore
increase in confidence of the network's predictions
this in $4. This increase in the network's 's confidence during training is key cause of miscalibration.
4 IMPROVING CALIBRATION USING FOCAL LOSS
As discussed above, overfitting on NLL, which is observed as the network grows more confident on all of its predictions irrespective of their correctness, is strongly related to poor calibration in neural networks. One cause of this phenomenon is that the cross-entropy objective minimises the difference between the softmax distribution and the ground-trun one-hot encoding over an entire mini-batch, irrespective of how well network classifies individual samples in the mini-batch. In this work, we study an alternative loss function called focal loss (Lin et al., 2017), which tackles this problem by weighting loss components generated from individual samples in mini-batch by how well the model classifies them. Focal loss for the 5th sample in the training set is computed as C, = - - pt.g.)' ) log Pt.y. where 7 2 0 is a user-defined hyperparameter. Focal loss attributes more importance to samples that the network misclassifies than to ones that are correctly classified. In this paper, we study properties of focal loss that help in calibration.
Empirical Observations: In order to analyse the behaviour of neural networks trained on focal loss, we use the same framework as mentioned above, and train four ResNet- 50 networks on CIFAR- 10, one using cross-entropy loss, and three using focal loss with metrics throughout the training period of these networks, for comparison. In Figures (a)), (b) and (c), we plot the average NLL over the test set, and for the correctly and incorrectly classified test samples, respectively, at each training epoch. In Figure 3()), we plot the softmax entropy for the misclassified test samples. In Figures 3(e) and (f), we plot the average probability value Pi.ys predicted for the ground-truth class, for correctly and incorrectly classified test samples, respectively. In Figure 3g)). we plot the test set ECE for all four models against the training epoch.
2, or 3. We then observe various
Figure 3(c) shows that in contrast to the network trained using cross-entropy, for which the NLL for misclassified test samples increases significantly after epoch 150, the rise in NLL for the networks trained on focal loss is much less severe. Moreover, from Figure 3(d), we notice that the softmax entropy for misclassified test samples is consistently (if marginally) higher for focal loss than for cross- cross-entropy. These observations indicate that the focal loss models are less confident about their incorrect predictions than the cross-entropy model. From Figure 3(f), we notice that the probabilities predicted for the correct classes for misclassified test samples are slightly higher for focal loss than for cross-entropy. Conversely, Figure 3(e) shows that the probabilities predicted for the correct classes for correctly classified test samples are somewhat lower than for cross-entropy. (Note that in Figure 3(b), the average test NLL values for the focal loss networks on the correctly classified samples also somewhat higher than for cross-entropy, which similarly indicates a slight decrease in confidence on those samples.) Nevertheless, for all four models, average confidence on correctly classified samples remains over 0.95 by the end of training, which is more than enough to make these
5

---

Under review as a conference paper at ICLR 2020
1.0
0.4
6.20
1.4
0:36
0.14
0.2
(p=3)
6:150
6.175
1.0
20.10
0.6
0.8
E0
20.00
5.06
2000
Baue
2 Ba.r
50:000
0.2
0.04
0.000
B.22
6:025
0.0
0.2
Propabiliy
0.4
0.5
0.8
1.0
0.00
Gradient
20
0.00
Grackent Norm
6000
Crudien horm
25
(a)
(b) Epoch 10
(c) Epoch 100
(d) Epoch 200
Figure 4: (a): g(p 7) vs p and (b-d): histogram of the gradient norms of the last linear layer for both cross entropy and focal loss.
predictions usable by downstream tasks. Furthermore, the classification errors on the test set for all four models are also almost the same (refer fifth row of Table 3). Given both of these points, along with the ECE plot in Figue 3()), it is thus reasonable to conclude that on a practical level, the models trained on focal loss more calibrated, whilst retaining both their classification accuracy and the majority of their confidence on correct predictions.
Theoretical Justification: As mentioned previously, once a model trained using cross-entropy reaches high accuracy on the training set, the optimisation algorithm may try to further reduce the training NLL by increasing the confidence values for the correctly classified samples, rather than focusing on the misclassified samples. One way it could achieve this would be to increase the weights of the network to increase the magnitudes of the logits. In fact, this hypothesis would help to explain the observation made in (Guo* et al., 2017) that models trained using some form of weight decay are better calibrated.
To verify this hypothesis, we plot the L2 norm of the weights of the last linear layer for all four networks as a function of the training epoch (see Figure 3(h). is interesting to note that although the norms of the weights for the models trained on focal loss are initially higher than that for the cross-entropy model, a complete reversal in the ordering of the weight norms occurs between epochs 150 and 250. In other words, the network starts to become miscalibrated, the weight norm for the cross- cross-entropy model also starts to become greater than those for the focal loss models. In practice, the reason for this is that focal loss, by design, starts act regulariser on the network's 's weights once the model has gained a certain amount of confidence in its predictions. To better understand this, we start by considering the following proposition (see the Appendix B for a proof).
aL, aL. g(pr. 7), Proposition For focal loss f and cross-entropy loss Lo the gradients Ow aw
). 7 2 0 is the focal loss hyperpa- where g(pr.n.)) 7) = (1 - "i..),,))" - TPia.s (1 - -i...) 7 DC1
isf il 7) E [0, 1]. To visualise the behaviour of p(p,7), we plot it against for various values of 7 in Figure 4(a). It is quite clear that g(P, 7) lies in [0, 1) for values of p that lie above a certain threshold pu- In the interval [0. Po). however, g g(p, 7) > 1. In fact, for given threshold po,
rameter, and w are the last layer network parameters. Thus,
aL. Sw
dw
efficiently compute (using the following proposition) a value 7- such that g(p, 7²) < 1 for p > Po (see the Appendix B for a proof).
W-1 ((1-a/b) log where a Po. po log po and W-1 1 the Lambert-Wi W function (Corless et al., 1996). Moreover, for p 2 Po and 7 2 >", the equality g(P,7) = 1 holds only for p = Po and 7 = 7
Proposition 2. For p 2 po. 9(P,)) 7) 1 for all y 2 7" = t + log a
Thus, given a Po. we can always use Proposition to compute 7 such that (i) g(po, 7*) = 1, (ii) for p < Po. "(, > 1, and (iii) for Po. g(P, 7²) < 1.
Implicit Regularisation: For a network trained using focal loss with a fixed y obtained using Po such that (Po, 7) = 1 then in the initial stages of training, when Pt,s < Po in general, ( 7) > 1 and the model gains confidence on its predictions faster than it would for cross-entropy. However, as soon as Pt.p, crosses the threshold po, g(pt...: 7) falls below and reduces the size of the gradient updates made to the network weights, thereby having a regularising effect on the weights. This is why, in Figure 3()), we find that the weight norms of the models trained with focal loss are initially higher than that for the model trained using cross-entropy. However, as training progresses, we find that the ordering of the weight norms reverses, focal loss starts regularising the network weights.
Figure 4(b-d) give further insights by plotting histograms of the gradient norms of the last linear layer (over all samples in the training set) epochs 10, 100 and 200, respectively. At epoch 10,
6

---

Under review as a conference paper at ICLR 2020
the gradient norms for cross-entropy and focal loss are similar, but as training progresses, those for cross-entropy decrease less rapidly than those for focal loss. Finally, note from Figure 4(a) that with higher values of % the fall in (P 7) gets steeper. We would thus expect the weight regularisation effect to be higher for models that use higher values of y. This explains why, of the three models we trained using focal loss, the one with y = 3 outperforms (in terms of calibration) the one with y 2, which in turn outperforms the model with = 1. Based on this observation, one might think that, in more calibrated model. However, this not the case, as we notice from Figure 4(a) that for 7 2 7, g(p..v..)) 7) reduces to nearly 0 for relatively low value of Pi.y. (around 0.5). As a result, using values of y that are too high will cause the gradients to die (i.e. reduce to nearly 0) early, at point at which the network's predictions remain ambiguous, thereby causing the training process to fail. To this end, we provide Proposition that allows us to efficiently get a cumpled-depender gamma in a principled manner. Experimentally we show that the sample-dependen r does provide the best result among all the approaches we compare with. We also provide empirical analysis of focal loss and cross entropy a linear model in Appendix G.
general, higher value of gamma would lead
5 EXPERIMENTS
We perform experiments on multiple image and document classification datasets to verify the effectiveness of focal loss for training calibrated models. For image classification experiments, we use CIFAR-10 10 (Krizhevsky, 2009) and CIFAR- 100 (Krizhevsky, 2009) and for document classification we use 20 Newsgroups and the Stanford Sentiment Treebank (SST) (Socher et al., 2013). We provide details on the datasets and the train/validation/tess splits for each dataset in the Appendix C. On CIFAR-10 and CIFAR-100, Wide-ResNet-26-10 10 (Zagoruyko and Komodakis, 2016), and DenseNet-121 (Huang et al., 2017).We We train the Global Pooling Convolutional Network (Lin al., 2014) on 20 Newsgroups and the Tree- LSTM (Tai et al., 2015) on the SST Binary dataset. We provide implementation details for each of the state of the art networks in Appendix D. For each dataset- netwsrt pair, we train the network using each of the following loss functions:
train the networks: ResNet-50,, ResNet- 1 10 (He et al., 2016 6),,
Baselines (Cross-Entropy, MMCE (Kumar et al., 2018)) and Brier Score (Brier, 1950): Models trained on cross-entropy loss. MMCE loss (i.e. cross-entropy with an MMCE regulariser) and Brier score serve as our baselines for comparison. MMCE (Maximum Mean Calibration Error) is continuous and differentiable proxy for calibration error and hence, can be directly optimised using standard optimisers like Stochastic Gradient Descent. Normally, MMCE is optimised as a regulariser alongside cross-entropy. Brier score, for a single sample, is computed as the squared error between the predicted softmax vector and the one-hot ground truth encoding. We find Brier score to be particularly relevant baseline for calibration there is a distinct penalty for increasing the confidence on incorrect classes. In addition, it is relevant as it can be decomposed into calibration and refinement (DeGroot and Fienberg, 1983).
Focal Loss (Fixed 7): We train various models each using three values of y set to 1, 2 and 3.
Focal Loss (Sample-Dependent 7): From Proposition 2, we know that for given probability p. we can find a value of 7 such that either g(p, 7) E [0, 1), and focal loss regularises the weights of the network, or 7( > 1, and focal loss speeds up the process of making the network more confident on the correct class. In this approach, we follow the idea of accelerating the rise in the value of p..y. (the confidence on the correct class y; for the jth training sample) as long as p..g. < 0.5. As soon as Pi.y, 2 0.5, we can confirm that the 7th training sample has been predicted correctly. Hence, when Pt.g. 2 0.5. we want to regularise the weights to prevent the network from becoming overconfident and miscalibrated. There can be multiple ways of choosing sample- dependent y that satisfy the above condition. However, with just few initial tries we found these two policies to produce quite competitive results on the validation sets: (a) Focal Loss (sample-wise 7 5.3,2): 7 = 5 for Plyy E [0, 0.1 (as (0..19,5) 5) se 1), 7 3 for Pi.. (0. 0.5) and y = 2 for Pt.y. E (0.5, 1], and (b) Focal Loss (sample-wise wise y 5,3): 7 = 5 for pi.. E [0, 0.19), and y = 3 for pr.,. E (0.19, 1].
Focal Loss (Scheduled 7): As simplification to the above approach, we also investigated the use of a schedule for 7 during training, as we expect the value of pt.,. to increase in general for all samples over time. In particular, we report results for two different schedules: (a) Focal Loss (scheduled y 5,3.2): 7 = for the first 100 epochs, 7 = 3 for the next 150 epochs, and 7 = 2 for the last 100 epochs, and (b) Focal Loss (scheduled 7 5,3,1): 1): 7 = 5 for the first 100 epochs, 7 3 for the next.
7

---

Under review as a conference paper at ICLR 2020
Dataset
Model
Cross
Eatropy Post
Brier Loss
MMCE
Focal Loss (sample-wise 7 5,3)
Pre
Pre
Post T
Pre
Post T
Pre T
Post
ResNet 50
12.52
3.42(2.10) 10)
6.52
3.64(1110) 10)
15.32
2.38(1.30)
4,50
2.00(1.10)
CIFAR-100
Wide ResNet 26-10
RosNet
19.05
15.33
4.43(2.50)
2.83/2220
7,88
4.31
2.70
(.65(1.2))
19.14
13.17
3.86(2.30)
4.37(1.90)
3.03
8.56
4.12(1.20)
164(1.10)
DonsoNet 121
20.98
4,27(2.30)
5,17
1.
19.13
3,06(2110)
3,73
1.310.10)
ResNet 50
4.35
1.35(2.50)
1.82
1.08(1 10)
4.56
1,192.60)
1.55
0.95(1.10)
CIFAR-10
Wide RoaNet 26-10
ResNet
3.23
0.92(2.20 20)
1.09(2 (0)
2.56
1.25
1.2541.20
1.25((100 (0))
5.08
3.29
0.86(2.20) 20))
1.42/2.0))
1.87
1.56
0.84(0.90)
1.07(1.10)
DenseNet 121
4.52
1.31(2.40)
1,53
1.53(1.0)
5.10
1.6(2.50)
1.22
1.22(1.00)
20 Newsgroups
Glohal Pooling CNN
17.92
2. 2.39(3.40)
13.58
3.22(2.30
15:48
6,78(2.20)
6.92
2.19(1.50)
SST Binary
Tre LSTM
7.37
2.62(1880)
9.01
2.79(2500)
5.03
4.02(1.50)
9.19
1.83(0.70)
Table 1: ECE (%) computed for different approaches both pre and post temperature scaling (cross- validating T on ECE). Optimal temperature for each method is indicated in brackets.
Dataset
Model
Cross Entropy Post
Brier Loss
MMCE
Focal Loss (sample-uise 5,3)
Pre
Post T
Pre
Post
ResNet
17.52
3.67(2.10)
6.52
1.6X))20)
15.32
2.441.80)
4.39
2.33(1.10)
CIFAR-100
Wide ResNet 26-10
ResNet
19.05
15.33
2.89 20)
5. 5.50 40)
4.22
7.73
2.81(1.10)
4.53(1 200
13.16
19.14
4,25(1.90)
4.85(2.30)
2.75
8.55
3.96(1.20)
166((110)
DenseNet 121
20.98
5.09(2.30)
5.04
2.56(L.10)
19.13
3.07(2.10)
3,55
1.24(1.10)
ResNet 50
4.33
2. 14(2.55)
1.74
1.23(1100)
4.55
2,0622550)
1.56
1.26(1.10)
CIFAR-10
Wide ResNet 26-10
ResNet
3.23
1.81(2.70)
1.69/2:20 20)
2.60
1.70
1.70 (02
1.63(0.90)
5.06
3.29
2.45(2.70)
1.692.10)
1.52
167(111)
1.38(0.90)
DenseNet 121
4,51
2 25(2500)
2,03
2. 281000)90)
5.10
2,46(2.40)
1.42
1.42(1.000
20 Newsgroups
Global Pooling CNN
17.91
2.20(3.30)
13:57
2.6002.00
15:21
7.07(2.10)
6.92
2.17(1.40)
SST Binary
Tree LSTM
7.31
2. (0((80)))
9.42
2.80(2.70)
4.49
3.72(1.50)
8.63
1.92(0.70)
Table 2: Adaptive ECE (%) computed for different approaches both pre and post temperature scaling (cross-validating T Adaptive ECE). Optimal temperature for each method is indicated in brackets.
150 epochs, and 1 for the last 100 epochs. We also tried various other schedules, but found these two to produce the best results on the validation sets.
We find the best policy for each of the above three focal loss variants by cross-validating on the validation set, and report its results in Table and We report results for all other policies in Appendix E.
DISCUSSION
Temperature Scaling: We report the ECE % (using 15 bins), both before and after temperature scaling, for focal loss (sample-dependan which performed the best) vs baselines in Table 1. Further, we report the same metric for different variants of focal loss in Table 4. Similarly we report AdaECE in Table 2 and Table 5. In f'ct our results show that in situations where ECE might imply that the model is well calibrated, AdaECE brings out the miscalibration present in the model. For example, in the case of WideResNet CIFAR10 for Cross Entropy, the best ECE obtained is 0.92, implying almost no miscalibration, whereas AdaECE is 1.69 showing further scope for improvements.
We compute the optimal temperature using two different approaches: (a) learning the optimal temperature by optimising NLL over a validation set, and (b) performing grid search over temperature values between 0 and 10, with a step of 0.1, and choosing the temperature that minimises the ECE/AdaECE over a validation set. We find the second approach to produce better results in general. Since we report ECE/AdaECE as the primary performance metric and grid search does not require a differentiable objective function, we directly minimise ECE/AdaECE over the validation set during grid search. Furthermore, since NLL is not a convex function of temperature, an optimisation algorithm may get stuck at a locally optimal temperature. We thus report the optimal temperatures and their corresponding ECEs/AdaFCES obtained using grid search (i.e. the second approach).
Performance Gains: It should be noted that for all the dataset-network pairs, we obtain of the-art classification accuracies, as shown in Table 3. For focal loss with a fixed 7, we found that 7 3 produced the best ECE results. This corroborates the observation we made in $4 that y 3 should produce better results than ? = or = 2, as the regularising effect for 7 = 3 is higher. For the sample-dependent 7 approach, we found the second policy (i.e. Focal Loss (sample-wise y 5,3) with y = 5 for Pi.y.. E [0, 0.19), and y = 3 for Pt.,, E (0.19, 1) to produce better results. Of the two approaches for scheduled 7, we found the first schedule (i.e. Focal Loss (scheduled 7 5,3.2) with
8

---

Under review as a conference paper at ICLR 2020
Dataset
Model
Cross Entropy
Brier Loss
MMCE
FL-3
FLS-532
FLA-53
ResNet 50
23,30
23.39
23.20
22.75
23.24
23,22
ResNet 110
22.73
25.10
23.07
22.92
22,96
22,51
CIEAR- 10O
Wide ResNet 26-10
20.70
20,59
20.73
19.69
20.13
20.11
DenseNet 121
24.52
23.75
24.00
23.25
23.72
22.67
ResNet 50
4.95
5.00
4.99
5.25
5.63
4,98
ResNet 110
4.89
5.48
5,40
5.08
5.71
5.42
CIFAR-10
Wide ResNet 26-10
3.86
4.08
3,91
4,13
4,46
4,01
DenseNet 121
5.00
5.65
5.46
20 Newsgroups
Global Pooling CNN
26.68
27.06
27.23
29.26
28.16
27.98
SST Binary
Tree LSTM
12.85
12.85
11.86
12.19
13.07
12.80
Table 3: Error (%) computed for different approaches, In this table, FL-3 denotes Focal Loss (fixed
3), FLS-532 denotes Focal Loss (scheduled y 5,3.2) and FLA-53 denotes Focal Loss (sample-wise
5,3) i.e. focal loss with sample-wise with y for Pt.y. E 0.199 and for Ps,ys (0.19, 1].
Dataset
Model
Focal Loss (fixed :; 3)
Focal Loss (scheduled 7 5,3,2)
Focal Loss (sample-wise 5.3)
Pre
Post T
Pre T
Post
Pre T
Post
ResNet
5.13
1.97(1110)
8.47
2.13(1.30)
4.50
2,00(1,00)
CIFAR- 100
ResNet 110
8.64
3.95(1.20)
11.20
3.43(1.30)
8,56
4.12(1.20)
Wide ResNet 26-10
2,13
2.13(1.00)
4.98
1.94(1.20)
3.03
164(1.10)
DenseNet 121
4.15
1.25(1.10)
7.63
1.96(1.20)
3,73
1.31(1.10)
ResNet 50
1.48
1.4((1.10)
2.97
1.53(1.20)
1.55
0.95(1.10)
ResNet 110
1.55
1.0((1.10)
3.33
1.36(1.30)
1.07(1110)
CIFAR-10
Wide ResNet 26-10
1.69
0.97(0.90)
1.82
144((110)
1.56
0.84(0.90)
DenseNet 121
1.32
1.26(0.90)
2.22
1.34(1.10)
1.22
1.22((100)
Newsgroups
Global Pooling CNN
3.51(1.50)
2.47(1.80)
2.19(1.50)
SST Binary
Tree LSTM
16.05
1.78(0.50)
2.64(0.90)
1.83/0.70)
Table ECE (%) computed for different approaches both pre and post temperature scaling (cross- validating T on ECE). Optimal temperature for each method is indicated in brackets.
Datasel
Model
Focal Loss (fixed 3)
Focal Loss (scheduled v 5,3,2)
Focal Loss (sample-wise 5,3)
Post
ResNet
5.08
2.35(1.20)
2-25(1.30)
2.33(1.10)
ResNet 110
8.64
4,14(1.20)
11.18
3.68(1.30)
8.55
3.96(1.20)
CIFAR-100
Wide ResNet 26-10
2.08
2,08(1 (.000)
2.11(1.200
2.75
1.63(1.10)
DenseNet 12.1
4.15
1.23(1.10)
7.61
2.04(1.20)
3.55
1.24(1.10)
ResNet
1.95
1.83(1.10)
2.95
2.18(1.20)
1.56
1,26(1.00)
ResNet 110
1.62
144(1.10)
3,32
1.91(1440
2.07
1.67(1.10)
CIFAR-10
Wide ResNet 26-10
1.84
1.54(0.90)
2.04
1.90X1.10)
1.52
1.38(0.90)
DenseNet 121
1.22
1.480090)
2.19
1.59X1.20)
1.42
1442(1.00
Newsgroups
Global Pooling CNN
8.65
3.78(1.50)
12.13
2.18(1.90)
6.92
2.17(1.40)
SST Binary
Tree LSTM
15.64
2.17(0.50)
2.94
2.50X0.90)
8.63
1.92/0.70)
Table 5: Adaptive ECE (%) computed for different approaches both pre and post temperature scaling (cross-validating T Adaptive ECE). Optimal temperature for each method is indicated in brackets.
5 for the first 100 epochs, 7 3 for the next 150 epochs, and y = 2 for the last 100 epochs) to produce better ECEs/AdaECESs for every model and dataset. We report the ECE values for the best performing approaches in Table and 4. Similarly we show the AdaECE values of these models in 2 and 5. Full results (ECE, AdaECE, MCE, NLL and test error) for all approaches are reported the Appendix E.
is clear from Table and 4 that focal loss with sample-dependent y outperforms all the baselines (cross-entropy, brier score and MMCE). It broadly produces the lowest ECE and AdaECE values both before and after temperature scaling. This observation is particularly encouraging, as indicates that principled method of obtaining values o for focal loss work well, thereby alleviating the need to naively cross-ved vaossval the hyperparaneter. Also, Table and Table 5 show that other focal loss based approaches also very competitive.

---

Under review as a conference paper at ICLR 2020
Dutaset
Model
Cross Entrogy (Pee T)
Cross Entraps (Pox T)
MMCE T)
MMCCE (Pox
Focal Less
Focal Loss (Post T)
159907
Accuracy
199905
Accuracy
15999%
Aocunay
199975
Accuacy
1599%
Accanay
199999
Accuracy
CIRAB-10 CIRAK-ED
RoNNa 110
97:11
11.5
96.
10,82
$8.83
89.51
91.10
68
ResNet
90.50
96.72
7.33
59.73
4.21
99.57
14:27
90.53
Table 6: Percentage of test samples predicted with confidence higher than 99% and the corresponding accuracy for Cross Entropy, MMCE and Focal loss computed both pre and post temperature scaling (represented in the table pre T and post T respectively).
Confident and Calibrated Models: It is interesting to note that for focal loss with sample-based Y (refer Tab. 1 and Tab. 2), beside most other focal loss models, the optimal temperatures are very close to 1, mostly lying between 0.9 and 1.1. By contrast, the optimal temperatures for the baselines (cross-entropy and MMCE) are significantly higher, with values lying between 2.0 to 2.8. An optimal temperature close to indicates that the model is innately calibrated and cannot be made significantly more calibrated by temperature scaling. Furthermore, an optimal temperature that is much greater than 1 can make the network underconfident in general, as its outputs temperature-scaled irrespective of their correctness. To see this, we follow the approach adopted in Kumar et al. (2018), and measure the percentage of test samples that predicted with a confidence of 0.99 or more (we call this set of test samples S99). In Table 6, we report |S99] as percentage of the total number of test samples, along with the accuracy of the samples in S99 for ResNect-0 and ResNet-1 10 trained on CIFAR-10, 10, using cross-entropy loss, MMCE loss, and focal loss. We observe that |S99] for the focal loss model is much lower than for the cross-entropy or MMCE models before temperature scaling. However, after temperature scaling, |599] for focal loss is significantly higher than for both MMCE and cross-entropy. The reason is that with an optimal temperature of 1.1, the confidence of the temperature-scaled model for focal loss does not reduce as much as those of the models for cross- cross-entropy and MMCE, for which the optimal temperatures lie between 2.5 to 2.8. We thus conclude that models trained on focal loss are not only more calibrated, but also better preserve their confidence on predictions, even after being post post-processed with temperature scaling. We present some qualitative results to support this claim in Appendix F.
7 CONCLUSION
In this paper, we have shown that training using focal loss can yield multi-class classification networks that are more naturally calibrated than those trained using more conventional cross-entropy loss. There are sound theoretical reasons expect this: in particular, as we show in $4, focal loss implicitly regularises the weights of a network during training, reducing NLL overfitting and thereby improving calibration. Extensive experiments on a variety of computer vision (CIFAR-IO/ Newsgroups/SST) datasets, and with a wide variety of different network architectures, show that this expectation is also borne out in practice. Our results show that almost all cases, networks trained with focal loss are more calibrated than those trained with cross-entropy loss, whilst having similar levels of accuracy, making their predictions much easier for downstream components to trust. We also introduce principled way of getting sample-dependent yy for focal loss and we show that it produces good results across datasets and models.
and NLP (20
REFERENCES
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 1950.
Chensi Cao, Feng Liu, Hai Tan, Deshou Song. Wenjie Shu, Weizhong Li, Yiming Zhou, Xiaochen Bo, and Zhi Xie. Deep Learning and Its Applications in Biomedicine. Genomics, Proteomics & Bioinformatics, 16(1): 17-32, 2018.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning in drug discovery. Drug Discovery Today, 23(36):2211122) 2018.
Robert M Corless, Gaston H Gonnet, David EG Hare, David Jeffrey, and Donald E Knuth. On the lambertw function. Advances in Computational mathematics, 5(1:332--359 1996.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 1983.
Chuan Guo Geoff Pleiss* Yu Sun*, and Kilian Q Weinherger. On Calibration of Modern Neural Networks. In ICMI., 2017.
10

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, 2017.
Andreas Kamilaris and Francese X Prenafcta-Boldú. Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 147:70-90, 2018.
Justin Ker, Lipo Wang. Jai Rao, and Tchoyoson Lim. Deep Learning Applications in Medical Image Analysis. IEEE Access, 6:9375-9389, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009,
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Measures For Neural Networks From Kemel Mean Embeddings. In ICML, 2018.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal Loss for Dense Object Detection. In ICCV, 2017.
Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Steeven Janay, and Christian Gagné. A New Loss Function for Temperature Scaling To Have Better Calibrated Deep Networks. arXiv preprint 1rX0v 1586v1, 2018.
Mahdi Pakdaman Naeini and Gregory F Cooper. Binary Classifier Calibration using Ensemble of Near Isotonic Regression Models. In ICDM, 2016.
Mahdi Pakdaman Nacini, Gregory F Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In AAAI, 2015.
Ng. 20 Newsgroups. https:: /gtthbb.///
thub. com/ avi ra kuka 22007/MCC// 2018. Accessed: 2019-05-
22.
Alexandru Niculescu-Mizz and Rich Caruana. Predicting Good Probabilities With Supervised Learning. In ICML,, 2005.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors word representation. In Empirical Methods in Natural Language Processing (EMNLP). 2014.
John Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances Large Margin Classifiers, 10(3):61-74 1999.
PyTorch-CIFAR. Train CIFARIO with PyTorch, 2019. ci faz (as of 22nd May 2019).
Available online https://github. com/
Avanti Shrikumar and Anshul Kundaje. Calibration with Bias-Corrected Temperature Scaling Improves Domain Adaptation Under Label Shift in Modern Neural Networks. arXiv preprini arXiv: 066552, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment trechank. In Empirical Methods in Natural Language Processing (EMNLP), 2013.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree- structured long short-term memory networks. Association for Computational Linguistics (ACL), 2015.
TTc.L.STM. FreelSTMe https://g/t wtw.. wwwp///////ttooocoocooo/oo 995 htts://eeell.S...ee... ment 2015. Ac- cessed: 2019-05-22.
Jinjiang Wang. Yulin Ma, Laibin Zhang, Robert Gao, and Dazhong Wu. Deep learning for smart manufacturing: Methods and applications. JMSY, 48:144-155 156. 2018.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In ICML, 2001.
Bianca Zadrozny and Charles Elkan. Transforming Classifier Scoes into Accurate Multiclass Probability Estimates. In SIGKDD, 2002.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks, In Proceedings of the British Machine Vision Conference (BMVC), 2016.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
In $ B, we show the proofs of the propositions formulated in the main text. We then describe all the datasets we used for experiments in C which is followed by implementation details of the state of the art networks in D.
B PROOFS
Here we provide the proofs of both the propositions presented in the main text. While Proposition helps us understand the regularization effect of using the focal loss, the Proposition 4 efficiently provides us the y values in principle way such that sample-dependent. Implementing the sumple-dependen
is very easy as implementation of the Lambert W function (Corless al., 1996)
available in standard libraries (e.g. python scipy). Using it we could get quite competitive results with just few initial sample- sample-dependent policies that we tried.
Proposition For focal loss L and cross-entropy loss Lo the gradients aL, &he g(pi ?), where g(pi. 7) = (1 - -P ) TPPias (1 - Dio,) log(pe,)) Doog.)""
"Bw "ow is the focal loss hyperpa-
rameter, and are the last layer network parameters. Thus, 8C,
ac. 8w
iff pf 7) E [0, 1].
8w
Proof. Let be the linear layer parameters connecting the feature map to the logit
Then, using
the chain rule, 8C,
Os
09..0, aC,
Ope..s Similarly, 8w
8C. =
0s
ac_
The
Ow čw
ös
8w
0s
derivative of the focal loss with respect to Pi the softmax output of the network for the true class ye. takes the form
1(11 -i.g)) VDi. (1 -
1
log (Pg(()..))
Opi.v:
Pius
aL.
0p.. g(pt... 7),
(4)
in which g( 7) = (1 -Po.p Pps.g (1 pr. ) log(ps.u.)
and
BC.
1
It is thus
Dp..
Pt.y.
straightforward to verify that if g(Dinn E [0, 1]. then aC,
S
8C. ap
which itself implies
that
&Cc Ow
Bw
loga W-1 a (1-11/0) log loga), where Po. po log po and W-1 is the Lambert-Ke W function (Corless et al., 1996). Moreover, for p P2 Po and 2 7", the equality g(p, 7) = 1 holds only for = Po and = =-"-
Proposition 4. For 2 po. g(p," 7) S 1 for all 2-
Proof. We derive the value of for which g(po, 7) for given po E [0. 1). From Proposition 4.1, we already know that
ac, aC.
ap..y: Opim g(pi.. 7),
(5)
where Cy is focal loss, L. is cross entropy loss, Pi.g. is the probability assigned by the model to the ground-truth -tuth correct class for the 5th sample, and
g(p 7) (1 (1--pin) Pi.,,)" TPi,,, (1 Pi.y,) log(pi.g)) logg
(6)
For p [0, 1], if we look the function g(p,7), 7), then and that g(p.7) 7) = 0 when p- = 1. To observe the behaviour of g(p, 7) for intermediate values of first take its derivative with respect p:
clearly see that g(p, 7)
0g(p,7) 7) 7 (1-p))~ 2(1 pp 101-011--0 log p + (7 - 1)p log p]
Op
In Equation 7, 7(1 2-2 > 0 except when = 1 (in which case 7(1 - --p --2 0). Thus, to 9e(p.7)
observe the sign of the gradient
focus on the term
Up
2(1 p) p) log p+ (> 1)p logp.
(8)
12

---

Under review as a conference paper at ICLR 2020
Dividing Equation 8 by (-logp), p), the sign remains unchanged and we get
kpp 7) = 2(1 - p)
logp + -pp
We can that k(p, 7) + 1 p and k(p,7) -1 (1 + 7) as p - 1 (using I'Hôpital's rule). Furthermore, k(p, 7) is monotonically decreasing for p E [0, 1]. Thus, as the gradient dg(p.c) Op monotonically decreases from positive value at = to a negative value at p = 1, we can say that 7) first monotonically increases starting from (as p - 0) and then monotonically decreases down to 0 (at = 1). Thus, for some threshold pa > 0 and for some y > 0, g(p, 7) = 1. then
vp > Po- g(,,7) 7) < 1. We now want find y such that Vp 2 po. (p. 1 1. First, let a (1 pp) and = Po po- Po- Then:
(Po. 7) (1 11-po po) (Po -po)> log Po
(1 - po) 1- Pa) - -Po log pol 1
a² (a - -b)) S
(v 1) + log(a -b) 5
( 6) log + log(a 7b) - (1 (1- 4 )
log
(10)
(a yb)e (-/b) loga a S a (1-0//)
5) (77-a/b) loga all a(l-a/b)
b
al (1-a/b)
( )) log a elt-a/blloge
log
where a = (1 Po) and = Po log Po- We know that the inverse of y = xe" is defined as x (y). where W is the Lambert-W function (Corless et al., 1996). Furthermore, the r.h.s. of the inequality in Equation 10 always negative, with a minimum possible value of - 1/e that occurs at Po 0.5. Therefore, applying the Lambert-W function to the r.h.s. will yield two real solutions (corresponding to principal branch denoted by Wo and negative branch denoted by 1). We first consider the solution corresponding to the negative branch (which is the smaller of the two solutions):
a11 (1-a// a/b)
(17
a T loga S SW-1 SWI(-
log
(11)
(1-a/b)
W-1(-
log
loga a
If we consider the principal branch, the solution is
(1-b/)
a b
Wo
log a
(12)
log a
which yields a negative value for that we discard. Thus Equation 11 gives the values of 7 for which p > Po. then g(p. 7) < In other words, g(po. 7) = 1, and for any p < Po, g(p, 7) > 1.
C DATASET DESCRIPTION
We use the following image- and document-classifiaation datasets in our experiments:
1. CIFAR-10 (Krizhevsky, 2009): This dataset has 60,000 colour images of size 32 x 32, divided equally into 10 classes. We use a train/validationfees split of 55,0000550000/1000
images.
2. CIFAR-100 (Krizhevsky, 2009): This dataset has 60,000 colour images of size 32 x 32, divided equally into 100 classes. (Note that the images in this dataset not the same images in CIFAR-10.) We again use train/validation/test split of 4550000/5,000100000 images.
3. 20 Newsgroups: This dataset contains 20,000 news articles, categorised evenly into 20 different newsgroups based their content. It is popular dataset for text classification. Whilst some of the newsgroups are very related (c.g. rec. rec.moooreseles and rec. rec.autos), others are quite unrelated (c.g. sei.space and misc.fore forsale).. We use train/validationttest split of 15,098/900/3 999 documents.
13

---

Under review as a conference paper at ICLR 2020
4. Stanford Sentiment Treebank (SST) (Socher et al., 2013): This dataset contains movie reviews in the form of sentence parse trees, where each node is annotated by sentiment. We use the dataset version with binary labels, for which 6,920/872/1,87 documents ing/validation/test split. In the training set, each node of parse tree is annotated as positive, neutral negative. At test time, the evaluation is done based on the model classification the root node, i.c. considering the whole sentence, which contains only positive or negative sentiment.
used the train-
D IMPLEMENTATION DETAILS
For training networks on the image classification datasets (CIFAR-10 and CIFAR-100) we use SGD with momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the first 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs. We use training batch size of 128. Furthermore, we augment the training images by applying random crops and random horizontal flips.
For 20 Newsgroups, we train the Global Pooling Convolutional Network (Lin et al., 2014) using the Adam optimiser, with learning rate 0.001, and betas 0.9 and 0.999. The code is Py adaptation of (Ng). We used Glove word embeddings (Pennington et al., 2014) to train the network. We trained all the models for 50 epochs and used the models with the best validation accuracy.
For the SST Binary dataset, we train the Tree-LSTM (Tai et al., 2015) using the AdaGrad optimiser with learning rate of 0.05 and a weight decay of 10 -4, as suggested by the authors. We used the constituency model, which considers binary parse trees of the data and trains a binary tree LSTM on them. The Glove word embeddings (Pennington et al., 2014) we used were also tuned for best results. The code framework we used is inspired by (TreeLSTM). We trained all of our models for 25 epochs and used the models with the best validation accuracy.
For all our models, we use the PyTorch framework, setting any hyperparameters not explicitly mentioned to the default values used in the standard models. For MMCE, we used À 2 for all the tasks, whilst we found À 8 to perform better for document classification. A calibrated model which does not generalise well to an unseen test set not very useful. Hence, for all the experiments, we set the training parameters in a way such that we get state-of-the-art test set accuracies on all datasets for each model.
E ADDITIONAL RESULTS
We use various metrics to compare the proposed methods based on focal loss with the baselines (i.e. cross-entropy, Brier score and MMCE).
We show all test ECE of all the focal loss models before and after temperature scaling in Table 7. Similarly, show all test AdaECE of all the focal loss models before and after temperature scaling in Table 8. We present the test set error for all the focal loss models in Table 9. We present the test NLL % before and after temperature scaling in Tables 10 and 11, respectively. We report the test set MCE % before and after temperature scaling in Tables 12 and 13, respectively.
In this section we use the following abbreviation to report results different varieties of Focal Loss. FL-1 refers to Focal Loss (fixed 1), FL-2 refers to Focal Loss (fixed 7 2), FL-3 refers to Focal Loss (fixed 7 3), FLS-531 refers to Focal Loss (scheduled y 5,3,1), FLS-532 refers to Focal Loss (scheduled 7 5,3.2), FLA-532 refers to Focal Loss (sample-wise 5,3,2) and FLA-53 refers to Focal Loss (sample-wise 7 5,3)
F QUALITATIVE RESULTS
In Figure 8, we present some qualitative results to show the improvement in the confidence estimates of focal loss in comparison to other baselines (i.e., cross entropy, MMCE and Brier score). For this, we take ResNet-50 networks trained on CIFAR-10 using all the four loss functions (cross entropy. MMCE, Brier score and Focal loss with sample-wise y 5,3) and measure the confidence of their predictions for four correctly and four incorrectly classified test samples. We report these confidences both before and after temperature scaling. It is clear from Figure 8 that for all the
14

---

Under review as a conference paper at ICLR 2020
Modei
FL.1
FL-3
FLS-531
FLS-332
ELA-332
FLA-33
Poot
Pee
1he
Phet
Poo
Post
Pee
Pee'
Pho
ioe
RoiNet 50
12.56
2.291150)
5.61
2.241 5m)
5.13
1.970,00
11:63
2.001 40)
9.47
2.100.30)
9.09
LSK1.30
4.50
2.001.10)
CHARI-00
Wale Bealee 26-10
8.03
2,5311440
4.64
2.901.0
2.13
21301000
9.36
2
4.99
L.9411200
2.501.200
3.00
Denocher 12:
14:26
2,90/1.00)
7:50
2.351
200
4.15
1,211.000
1335
2.001.50 240
7.63
1.9611200
8.34
2.351 300
535
1.300.00 10)
RoiNe: 50
3/42
10811.00
2.36
0.911 20
1.48
LA20100)
4.06
1.541.00)
2.07
1.501.20)
2.52
0.5911.30
1.55
0.50110)
1.501.00)
1891110
Wale Bealee
14611.30
1.42
1.001110
09918995
2.81
0L9611440)
1.82
LA501.00)
0.571.10)
1.56
0.540.00
Danocher 12:
3.44
1.0233440
1.85
10001000) (0)
1.32
1.26/0/90)
6.32
1.00) 100
2.22
1.9401.000
2.45
1.3101.20 20)
1.22
1:221.100
20 Nenigauapo
Global Pooling CSN
15.06
2.142.60)
12.10
3.231 (00
9.07
3.511.50
13.55
4.3311.00
12:13
2.700.80)
12.20
2.5922 00)
6.92
2.191.001
SST Bisacy
LSTM
3.291.00
3.05
30000000
1.780050)
3.361140)
3.91
2.6400000
2.7700000
9.19
1.300.70
Table 7: ECE (%) computed for different focal loss approaches both pre and post temperature scaling valiiaani T on ECE). Optimal temperature for each method is indicated in brackets.
Dotact
Model
FL-1
Fl.-2
F1.3
FL5-531
FLS-533
FLA-532
FLA-53
Piz V
Pout
T
Poal
Psc
Pool
Prc I
Poat7
Poal
Prc 2
Peat
Prc
Pout
ReiNe: 50
12:86
2.5423.50 500
5.55
2.44C1
2,5511.20
11.55
2.0101
5.41
2.2511 30)
1.911 30)
6.39
2.351
1.811-30)
Wale Realeet 26 10
9.93
2.74:3.40)
4.65
2.561 20)
2.03
2,0801.00
9.20
2.5011 (U0)
5.00
2.1161.20)
5.00
2.5841.20 210
2.75
1.651. 110
Demochet 12)
14.24
2.2111.50)
7.50
2.161.20
4.15
1,2311.00
13.01
2.180.500
7.61
2,0411.20)
8:04
2.1001.30
3.55
1.241110)
RoNNe
1,5123.60
2:37
1.831.00
4.06
2.161
2.95
2.1811.200
1.251 30)
1.56
1.501
CIRAK-10
Wale Realet 26 10
ReaNet
2.70
3
1.5711.80
1.52:3.40
2.60
1.64
1.471.10)
1.3N1300
1.84
1,411,05
1,5410.90)
2.75
2.6211.00)
1.5701.50 55)
3.32
2.84
1.9111440)
1.9001.10)
1.68
1,4911 10)
1.541.00)
2.07
1.52
L.51110)
1.2800.00
Demochet 12)
3.44
1.851140)
1.50
1,291110
1.22
1,480/90)
4.31
2.20x1.500
2.19
1.9911220)
2.44
1.501 20)
1.42
142)100)
Newsgroops
Global Poiling CNN
15.06
2.55:2.50)
12.10
3.331
3,7811.50)
13.55
3,550
12.15
2,1811
12.19
5,141 700
6.90
2.171
Hinary
1.001.30
2,8000000
Table 8: AdaECE (%) computed for different focal loss approaches both pre and post temperature scaling (cross-validating T on AdaECE). Optimal temperature for each method is indicated in brackets.
Dataset
Model
FL.-1
FI.-2
FL-3
FLS-531
FISS-532
FLA-532
FLA-53
ResNet 50
22.80
23.15
22.75
23,49
23.24
23.55
23,22
ResNet 110
22.36
22.52
22.92
22.81
22.96
22.93
22.51
CIFAR-100
Wide ResNet 26-10
19.61
20.01
19.69
20.13
20.13
19.71
20.11
DenseNet 121
23.82
23.19
23.25
23.69
23.72
22.41
22.67
ResNet 50
4.93
4.98
5.25
5.66
5.63
5.24
4.98
ResNet 110
4.78
5.06
5.08
6.13
5.71
5.19
5.42
CIFAR-10
Wide ResNet 26-10
4.27
4.27
4.13
4.11
4,46
4.14
4.01
DenseNet 121
5.09
4.84
5.33
5.46
5.65
5.46
5.46
20 Newsgroups
Global Pooling CNN
26.13
28.23
29.26
29.16
28.16
27.26
27.98
SST Binary
Tree LSTM
12.63
12.30
12.19
12.36
13.07
12.30
12,80
Table 9: Error (%) computed for different focal loss approaches.
Datasct
Model
Cross Eatropp
Hrice Loss
MMCE
FL-1
F1.-2
FL.33
FL.8-531
FLS85532
FLA-532
FLAA-53
ResNet 50
653.67
99.63
125.28
105.61
92.82
87.52
100.09
92.66
94.10
83.03
ResNett L10
17921
110.72
180.54
114.18
96.74
90.90
112.46
95.85
97.97
$9.92
CIFAR-100
Wilk ResNer 26-10
$4.62
119.58
87.56
77.80
74.66
83.61
78.52
76.92
DenseNeet 12)
205.61
98.31
166.65
115.50
93.11
87.13
107.91
93,12
91,14
85.47
ResNet 50
41.21
18.67
44.83
22.67
18.60
18:43
25:32
20.50
18.69
17.55
ResNet 110
47.51
20.44
55.71
22.54
19.19
17.80
32.77
22.48
19.39
18.56
CUPAR- 10
ResNet 26
13:85
28.47
17.66
14.96
15.20
15.37
14.55
DenseNiet 121
42.93
19.11
52,14
22,50
17.56
18.02
27.41
19.50
20,14
18.39
20 Newsgroups
Global Pooling CNN
176.57
130.41
158.70
140.40
115.97
109.62
128.75
123.72
124.00
109.17
SST Rinaay
Tice LSTM
50.20
$4.96
37.28
$3.90
47.73
50.29
50.25
53.13
45.08
49.25
Table 10: NLL (%) computed for different approaches pre temperature scaling.
correctly classified samples, the model trained using focal loss has very confident predictions both pre and post temperature scaling. However, on misclassified samples, we observe a very low confidence for the focal loss model. The ResNet-50 network trained using cross entropy is very confident even
15

---

Under review as a conference paper at ICLR 2020
Dafaset
Model
Cruss Entropy
Brier Loss
MMCE
FL-1
FL-2
FL-3
FLS-531
FLS-532
FLA-532
FLA-53
ResNer
106.83
99.57
101.92
94.58
91.30
57.37
92.17
91.58
92.83
88.27
ResNet
104.63
111.81
10673
80.92
93.73
Wide ResNet
97.10
65.77
95.02
CIFAR-100
DenseNet 121
119.23
98.74
113.24
100.81
91.35
87.55
98,16 16
91.55
90.57
86.06
ResNer.9
20.38
18:36
21.58
17.56
17.67
18.34
19:93
19.25
17.28
17.37
ResNet 110
21.52
19.00
24.61
17.32
17.53
17.62
23.79
20.21
17.78
18.24
CIFAR-10
Wide 26-10
15:85
16.
14,85
15.81
15.38
14.23
DenseNet
21.77
17.21
21.65
18.39
20 Newsgroups
Global Pooting CNN
87.95
93.11
99.74
87.24
93.60
94.69
97.89
93.66
91.73
93.98
SST Binary
Tre LSTM
41,05
38.27
36.37
45.67
47.72
45.96
45.82
54,52
45,36
49.89
Table 11: NLL (%) computed for different approaches post temperature scaling (cross validated on ECE).
Dataset
Model
Cross Entropy
Brier Less
MMCE
F1.11
FL-2
FL-3
FLS-531
FLS-5332
FLA-532
FLA-53
ResNet 50
44.34
36.75
39:53
33.22
21.03
13.02
26.76
23.56
22.40
16.12
Restet
51.24
22.57
Wide ResiMer 26-10
10.17
DenseNer 121
56.28
15.47
49.97
35.45
21.70
11.61
38.68
18.91
21.34
9.68
ResNer 50
38.65
31,54
60.06
31.75
25.00
21.83
30.54
23.57
25.45
14.89
ResNer 110
44.25
25.18
67:52
73.35
25.92
25.15
$4.18
30.38
30.80
18.95
CIFAR-10
Wide ResiNet 26-10
48.17
77.15
36.82
29.17
30.17
23.86
37.57
18.51
74,07
DenseNet
13.36
20 Newsgroups
Global CNN
36.91
31.35
34.28
26.02
24.29
17.44
SST
Tree LSTM
71.08
92.62
68.43
95.48
86.21
22.32
76.28
86.93
80.85
73.70
Table 12: MCE (%) computed for different approaches pre temperature scaling.
Dalaset
Model
Entropy
F1.S-531
F1.5-532
FLA--532
FLA--53
ResiNer
21.61
7.46
27.18
ResiNer 110
22.65
13.56
19:33
20.13
12.00
13.066
18.28
13.72
15.80
10.94
Wide Res|Net 26-10
14.18
13:42
16.50
10.28
18.32
9.96
13.18
11.01
12.50
9.73
CIBAR- 100
DenseNet 121
21.63
8.55
13.02
10.49
11.63
6.17
6.21
6.18
9.41
5.68
ResNei
20.60
22,46
23.60
23.86
28.17
15.76
22.05
23.85
24.76
26.37
CIBAR-10
Wide ResiNet 26-
26.63
77.15
74.38
25.64
28.63
20.23
36.56
DenseNer 121
32.52
19.39
27.03
19.68
22.71
76.27
21.05
32,76
35.06
13.36
20 Nowsgroups
Global Pooking CNN
6.10
13.04
15.12
7.79
11.88
22.22
16.86
11.85
8:37
12.65
SST Binary
Tiee 1STM
$8.48
91.86
$2.92
87.77
86.21
74.52
$4.27
88.85
82.42
76.71
Table 13: MCE (%) computed for different approaches post temperature scaling (cross validated on ECE).
on the misclassified samples, particularly before temperature scaling. Apart from focal loss, the only model which has relatively low confidences on misclassified test samples is the one trained using Brier score. These observations support our claim that focal loss produces not only a calibrated model but also one which is confident on its correct predictions.
G FOCAL LOss AND CROSS ENTROPY ON A LINEAR MODEL
To understand the effect of focal loss on simpler to analyse models, we conducted experiments generalised linear model on a simple data distribution.
Setup We consider a binary classification problem. The data matrix X E R2xN created by assigning each class two normally distributed clusters such that the mean of the clusters are linearly separable. The mean of the clusters are situated on the vertices of a two-dimensional hypercube. Further to that, for 10% of the data points, the labels were flipped. The model consists of a simple 2- 2-parameter logistic regression model. fw = a ow 21 + W222). We train this model using both cross entropy and focal loss with 1.
16

---

Under review as a conference paper at ICLR 2020
200
175
CE
Norm OT VV Weiggtts
1.75
Norm of Logits
180
FL
125
1.25
100
0.75
0.25
CE
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
Iterations
Iterations
Figure 5: (a): Norm of logits (b): Norm of weights.
Weight Magnification We have argued that focal loss implicitly regularizes the weights of the model by providing smaller gradients as compared to cross entropy. This helps in calibration as all the weights are large, the logits are large and thus the confidence of the network is large on test points. When the model misclassifies, it misclassifies with high confidence. Figure 5 shows, for generalised linear model, that the norm of the logits and the weights of a network blows for Cross Entropy compared to Focal Loss.
High Confidence for mistakes Figure 6 (b) and (c) shows that rumning gradient descent with cross entropy (CE) and focal loss (FL.) both gives the same decision regions i.e. the weight vector points in the same region for both FL and CE. However, as we have seen that the norm of the weights much larger for FL compared to CE, we would expect the confidence of misclassified test points to be large for CE as compared to FL. Figure (a) plots a histogram the confidence of the misclassified points and it shows that CE misclassifies almost always with greater than 90% confidence whereas FL misclassifies with much lower confidence.
18
12
Confidenos
(b)
(c)
Figure 6: (a): Confidence of misscfaasss (b): Decision boundary of linear classifier trained using cross entropy (c): Decision boundary of linear classifier trained using focal loss
H FOCAL LOSS MINIMISES REGULARISED BREGMAN DIVERGENCE
In this section, show that just like Cross Entropy minimizes upper bound on the KL-Divergence between the true distribution p and the predicted distribution focal loss minimizes a regularized KL divergence between p and q. We use (p. and CF q) to denote the focal loss with parameter
17

---

Under review as a conference paper at ICLR 2020
y and cross entropy between p and q respectively and K to denote the number of classes.
(I,(():))=-- = Σ ps log qi
K
2
Σ
7 )ps log 9
By Bernoulli's inequality Vy 2 and S q S
K
log -
Σ
Pid log 9i
Vi log 9 S 0
Σ pi log 9:
max Pj
lq log qil
By Holder's inequality with p 00,9
3
2 Σ
Pi log q: + +
go log
Pi S
CE ,Entropy())
We know that
KL (q) = CE(P. 9) Entropy(p)
(L (g) S "Entropy FL(p, Entropy (p)
FL(p.9) 2 (T(pllg) + Entropy "Entropy(g)
Experimentally, we found the solution of the Cross Entropy and Focal loss equation, i.e. the value of the predicted probability ý which minimizes the loss, for various values of p binary classification problem (i.e. K = 2) and plotted it in Figure 7. Note how Focal loss favours a more entropic solution q that closer 0.5. In other words, as Figure 7 shows, solutions focal loss (Eqn equation 13) will always have higher entropy than that of Cross Entropy.
q = argmin, (1 x x x² (1 - p) log (1 -x) z) 0<
(13)
1.0
CE
FL-1
0.8
FL-3
0.4
0.2
p
Figure 7: Optimal for various values of p and y
18

---

Under review as a conference paper at ICLR 2020
Actual Label: Cat
Actual Label: Ship
Actual Lebel: Airplane
Actual Label: Frog
Cross Entropy
Prodicted Label: Cat Confidunce (Pre T): 0.9099 Confidence (Pest Th: 0,9880
Prodicted Label: Ship Confidince (Pre T): 0.8900 Confidencs (Post Th: 0,9019
Con fidemon (Pre T): 0.000o Confilencs (Post Ty 9893
Predicted Label: Airpiane
Confidence (Pre Tl: 0.9000 Confelence (Post T): 9863
Predicted Label: Frog
Actual Label: Cat
Acqual Label: Ship
Actual Label: Airplace
Actual Label: Frog
Score
Confidence (Pre T): 0.9977
Prodicted Label: Cat
Confidence (Pre Ty: 0.8989
Prodicted Label: Ship
Confidance (Pre T): C. 6.996!
Predicted Label: Airplane
Confidence (Pre T): 0.9955 Confidence (Pust T): 0.9911
Prodicted Label: Frog
Confidence (Post T): 0.9953
Confidence (Post T): 0.9976
Confidence (Post Ty: 0.9923
Actuaa Label: Cat Prodictad Tabel: Cat Confidence (Pre 0,9999
Acuual Label: Ship
Actual Lebel: Airplane
Actual Label: Frog
MMCE
Confidence (Pre Ty: 0.9999 Confidence (Pre T): 0. 0.9999
Prodicred Tabel: Ship
Predicted Labell: Airplane
Predicted Label: Frog Confidence (Pre T.) 0.9939 Confidence (Pust 0.9732
Confidence (Post T): 0.9796
Confideacee (Post T): 0.9091
Confidencee (Post 0.9894
Actual Label: Cat Predicted Label: Cat
Actual Label: Ship
Actual Label: Airplane Predicted Label: Airplane
Actual Label: Frog
Predicted Label: Ship
Preducted Label. Frog
Focal Loss
Contidence (Pre 1): 0,9984
Confidence (Pre 1): 0.9977
Couhidence (Pre T): 0.9937
Condideace (Pre T): U.9954
Condidence (Post T): 0.9867
Confidence (Post T): 0.9956
Confidence: (Post Tl: 0.9884
Confidence (Post T); 0,8912
Actual Lebel: Dogg
Actael Label: Airpleue Predicted Label: Ship Confidence (Pre T): 0.9973
Actuol Label: Automobile Predicted Label: Track
Actual Label: Cal Predicted Label: Airplano Confidence (Pre Tr: 0,9966 Confideace (Post T): 0.8019
Predicted Label: Bird Confidence (Pre T): 0,9872
Cross Entropy
ConÉdence (Pre T): O 0.9999
Conáidecce (Post T): 0.7293
Confidence (Post T): 0.8703
Confidencee (Post T): 0.9611
Actual Label: Dag
Actuol Label: Airplone
Actuol Label: Automobille
Artuel Label: Cat Predicted Label: Airplane Confédance (Pre 17: 0.3753
Brier Score
Confidencee (Pre Tj: 0,775B
Predicted Label: Cet
Predicted Label: Shig Confidecc (Pre T): 0.9750
Combdence (Pre T): 0.9519
Predicted Label: Track
Confidence (Post T): 0,7396
Coufidence (Post T.: 0.9614
Confidence (Post T): 0.9338
Contideace (Post T): 0.3572
Actual Label:
Actuol Label: Airplane Predisted Label: Ship Confidence (Pre 0.9899
Actual Label: Automobile Predicted Label. Track Confdence T): Confideace
Actual Label: Predicted Label), Airplane Confidince Confidence
Predicted Label: Cat Confidence (Pre 0.9409 Confidince (Past 0.640G
MMCE
0,9936 0.7074
Coufidence (Post 0.3733
0.9124
Actual Label: Dog
Actual Lahels Airplane
Actual Lahels Autonobilc Predicted Label: Truck Confidence (Pre T): 0.5789
Actual Label: Cla
Predicted Label: Bird Con Renfidence (Pre T): 0.6714
Predicted Label: Slip
Prodicted Labels Airplane
Focal Loss
Confidence (Pre Ty: 07:00.25 0.7975
Confidence (Pre T): 0.4178 Confidence (Post 0.3937
Confidences (Post T): 0.6468
Confidence (Post T): 0.7740
Confidence (Post 0.5689
Figure 8: Qualitative results showing the performance of Cross Entropy, Brier Score, MMCE and Focal Loss (sample-wise 7 5,3) for ResNet-50 trained CIFAR-10. The first row of images have been correctly classified by networks trained on all four loss functions and the second row of images have all been incorrectly classified. For each image, present the actual label, the predicted label and the confidence of the prediction both before and after temperature scaling.
19


Reviews:
[{"review_id": "r1gDtRmDFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies the effect of the focal loss, proposed by Lin et al. in 2017 on network miscalibration, which appears when the network's confidence in its prediction does not match its correctness. The authors provide a theoretical explanation to the superior results of the focal loss for calibration. The temperature scaling technique of Guo et al. 2017 is applied (dividing the network's logits by a scalar learnt on a val set prior to softmax) to networks trained using the focal loss, with different options for the focal parameter, as well as the standard multi-class cross entropy and a few others. The experiments on CIFAR10/100 as well as two text dataset (20 Newsgroups, Stanford Sentiment Treebank) reach lower expected calibration error compared to the cross entropy (75% of relative improvement on cifar100 for instance).\n\nThe importance of the contribution will probably be discussed here. At first glance, it seems that the works build mainly on advances from Lin et al & Guo et al, but the authors do a promising job in combining the two.  \n\nPositive aspects: \n- The paper is well written. \n- Experiments on both image and text dataset demonstrate the superiority of the focal loss on several calibration metrics. \n- The theoretical explanation is convincing.   \n\nNegative points:\n- The importance of the problem is motivated by future assessments by downstream tasks but do not address this aspect in the experiments. In particular, as the images experiments are conducted on tiny images, an experiment on a real size image dataset would strengthen the paper. \n- The policy that works best for defining the sample wise tuning of the focal parameter was hand-made but ultimately uses only 3 parameters so finally it is not so bad.  \n\nMinor: \n- It'd be nice to illustrate the confidence improvements on a few qualitative examples, maybe in appendix.\n- 10 pages is too much (given that were were given instructions to be more severe with long paper) table 6 and 3 could be merged for instance. \n- The focal loss column results of table 1 should be the same as Table 5 (sample wise)?\n- could specify what MMCE means\n- clean the bibliography\n\nI've read the other reviews and authors' responses. Experiences on Tiny ImageNet are better than CIFAR but still a little far from what I'd call real images but I understand it can be difficult to run experiments on ImageNet. Since the choice of gamma seems to be leading consistent results also on tinyIN, I find it less concerning.    ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkePAgQ9KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes how the use of the now-standard focal loss can lead to improved calibration results when used to fit deep-models. When fitting a large capacity model with NLL, the model can often try to drive its predictions close to 1 (i.e. infinity pre-softmax) on the training set, ultimately leading to poorly calibrated models and overfitting behaviour. The focal loss appears to mitigate this issue.\n\nThe approach is extremely simple to implement, the theoretical justifications are believable, and the calibration/accuracy performances seem to be good -- for this reasons, I think that the paper should be accepted.\n\n(1) it would be interesting to compare the approach to using the standard cross-entropy applied to smoothed labels (i.e. (1-eps,eps) instead of (1,0) in binary classification and obvious generalisation in multi-class setting).\n\n(2) data-augmentation often greatly helps with calibration -- the paper did not describe in details what has been done on that front for the numerical investigations.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJlkRDV6ur", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores how focal loss can be used to improve calibration for classifiers. Focal loss extends the cross-entropy loss, which is -log(p_label), with a multiplicative factor equal to (1 - p_label)^gamma. Intuitively, this downweights the loss for elements where the probability of the correct label p_label is close to 1, relatively increasing the weight of the misclassified examples.\n\nSomewhat surprisingly, this tends to improve the calibration of the model. I say surprisingly because the focal loss is not a bregman divergence for all values of alpha so in general the expected minimizer of the focal loss for a fractional label is not the fractional label (i.e. the minimizer wrt x of - p (1-x)^gamma log(x) - (1-p) x^gamma log (1 -x) is not in general p).\n\nThe paper shows somewhat thorough experiments on many datasets justifying this observation, but the theoretical part is rather weak since it doesn't seem to address this issue with the focal loss.\n\nIt's also not very clear from reading the paper what the p0 should be when using the rule to automatically select the gamma of the focal loss.\n\nI'd support accepting the paper if the calibration properties of the focal loss itself was better analyzed on a simpler setup (linear models, or single parameter models) so it's easier to understand how it's helping calibration in the deep network setup and if the algorithm for choosing per-example gammas was more clearly stated out.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the effect of focal loss on calibration of neural nets.

On one hand, the reviewers agree that this paper is well-written and the empirical results are interesting. On the other hand, the reviewers felt that there could be better evaluation of the effect of calibration on downstream tasks, and better justification for the choice of optimal gamma (e.g. on a simpler problem setup).

I encourage the others to revise the draft and resubmit to a different venue.  

Decision: Reject