Title: THE INTRIGUING EFFECTS OF FOCAL LOSS ON THE CALIBRATION OF DEEP NEURAL NETWORKS Anonymous authors Paper under double-blind review

Abstract: Miscalibration -a mismatch between a model's confidence and its correctness -of Deep Neural Networks (DNNs) makes their predictions hard for downstream components to trust. Ideally, we want networks to be accurate, calibrated and confident. Temperature scaling, the most popular calibration approach, will calibrate a DNN without affecting its accuracy, but it will also make its correct predictions underconfident. In this paper, we show that replacing the widely used cross-entropy loss with focal loss allows us to learn models that are already very well calibrated. When combined with temperature scaling, focal loss, whilst preserving accuracy and yielding state-of-the-art calibrated models, also preserves the confidence of the model's correct predictions, which is extremely desirable for downstream tasks. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to theoretically justify the empirically excellent performance of focal loss. We perform extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (SST, 20 Newsgroup) datasets, and with a wide variety of different network architectures, and show that our approach achieves state-of-the-art accuracy and calibration in almost all cases.

1. INTRODUCTION:

Deep neural networks have dominated computer vision and machine learning in recent years, and this has led to their widespread deployment in real-world systems (Cao et al., 2018; Chen et al., 2018; Kamilaris and Prenafeta-Boldú, 2018; Ker et al., 2018; Wang et al., 2018) . State-of-the-art networks achieve high levels of accuracy for many tasks. However, many current multi-class classification networks in particular are poorly calibrated, in the sense that the probability values that they associate with the class labels they predict for different test samples overestimate the likelihoods of those class labels being correct in the real world. This is a major problem, since if networks are routinely overconfident, then downstream components cannot trust their predictions. The underlying cause is hypothesised to be that these networks' high capacity leaves them vulnerable to overfitting on the negative log-likelihood (NLL) loss they conventionally use during training (Guo* et al., 2017) .

Given the importance of this problem, numerous suggestions for how to address it have been proposed. Much work has been inspired by early approaches from the pre-deep learning era such as Platt scaling (Platt, 1999) , histogram binning (Zadrozny and Elkan, 2001) , isotonic regression (Zadrozny and Elkan, 2002) , and Bayesian binning and averaging (Naeini et al., 2015; Naeini and Cooper, 2016) . As deep learning has become more dominant, works have begun to directly target the calibration of deep networks. For example, Guo et al. (Guo* et al., 2017) have popularised a modern variant of Platt scaling known as temperature scaling, which works by dividing a network's logits by a scalar T > 0 (learnt on a validation subset) prior to performing softmax. Temperature scaling has the desirable property that it can improve the calibration of a network without in any way affecting its accuracy. Mozafari et al. (Mozafari et al., 2018) noted the downsides of using cross-entropy loss with temperature scaling, and proposed an alternative loss called Attended-NLL that helps temperature scaling achieve better calibration. More recently, Shrikumar and Kundaje (Shrikumar and Kundaje, 2019) have proposed an extension to temperature scaling that adds class-specific bias parameters to help eliminate systematic bias when performing domain adaptation. Separately, Hendrycks et al. (Hendrycks et al., 2019) have studied the effects of pre-training (vs. training from scratch) on model robustness and uncertainty: they make the interesting observation that because long periods of training can cause a network to become miscalibrated, tuning a pre-trained network, which facilitates faster convergence, can seemingly lead to a more calibrated model. Notably, since their approach complements temperature scaling, the two techniques can also be used together to achieve even better calibration overall.

Whilst temperature scaling's simplicity and effectiveness have made it a popular and state-of-the-art network calibration technique, it does have downsides. For example, whilst it scales the logits to reduce the network's confidence in incorrect predictions, this also slightly reduces the network's confidence in predictions that were actually correct. By contrast, Kumar et al. (Kumar et al., 2018) eschew temperature scaling altogether in favour of minimising a differentiable proxy for calibration error at training time, called Maximum Mean Calibration Error (MMCE). However, they also use temperature scaling as a post-processing step to obtain better results than cross-entropy followed by temperature scaling (Guo* et al., 2017) .

In this paper, we propose a technique for improving network calibration that works by replacing the cross-entropy loss conventionally used when training multi-class classification networks with the focal loss proposed by Lin et al. (Lin et al., 2017) for dense object detection. Since focal loss, as shown in §4, is dependent on a hyperparameter, γ, which needs to be cross-validated, we also provide a theoretically justified way to choose γ automatically for each sample and show it to outperform all the baseline models.

Informally, the intuition behind using focal loss is to direct the network's attention during training towards samples for which it is currently predicting a low probability value for the correct class, since trying to reduce the NLL on samples for which it is currently predicting a high probability value for the correct class is liable to lead to NLL overfitting, and thereby miscalibration (Guo* et al., 2017) . More formally, we show in §4 that focal loss can be seen as implicitly regularising the weights of the network by causing the gradient norm to be lower than it would have been with cross-entropy loss as training proceeds, which we would theoretically expect to reduce overfitting and improve the calibration of the network. In §5, we perform extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (20 Newsgroups/SST) datasets, and with a wide variety of different network architectures (e.g. ResNet-110/50, Wide-ResNet, DenseNet), to show that this is indeed the case.

Our experiments show that in almost all cases, DNNs trained with focal loss are more calibrated than those trained with cross-entropy loss, MMCE, and Brier loss Brier (1950) . Moreover, since our approach, like that of (Hendrycks et al., 2019) , is complementary to the temperature scaling, significant improvements in calibration over temperature scaling alone, and state-of-the-art results, can be achieved by training with focal loss and then performing temperature scaling.

2. PROBLEM SETUP:

Let D = (x i , y i ) N

i=1 denote a dataset consisting of samples from a joint distribution D(X , Y), where for each sample i, x i ∈ X is the input and y i ∈ Y = {1, 2, ..., K} is the ground-truth class label. Let pi,y = f θ (y|x i ) be the probability that a neural network f with model parameters θ predicts for a class y on a given input x i . The class that f predicts for x i is computed as ŷi = argmax y∈Y pi,y , and the predicted confidence as pi = max y∈Y pi,y . Note that the confidence, by definition, does not depend on the ground-truth label. The network is said to be perfectly calibrated when, for each sample (x i , y i ) ∈ D, the confidence pi is equal to the model accuracy P(ŷ i = y i ), i.e. the probability that the predicted class is correct. For instance, of all the samples that a perfectly calibrated neural network classifies with a confidence of 0.8, 80% should be correctly predicted.

One of the most popular metrics used to measure model calibration is the expected calibration error (ECE) (Naeini et al., 2015) , defined as the expected absolute difference between the model's confidence and its accuracy, i.e. E (xi,yi)∼D |P(ŷ i = y i ) -pi | . Since calculating P(ŷ i = y i ) is infeasible, the ECE cannot be directly computed. A workaround is to divide the interval [0, 1] into M equispaced bins, where the i th bin is the interval i-1 M , i M . Let B i denote the set of samples with confidence scores belonging to the i th bin. Then the accuracy A i of this bin is computed as

A i = 1 |Bi| j∈Bi 1 (ŷ j = y j )

, where 1 is the indicator function, and ŷj and y j are respectively the predicted and ground-truth labels for the j th sample. Similarly, the confidence C i of the i th bin is computed as C i = 1 |Bi| j∈Bi pj , i.e. C i is the average confidence of all samples in the bin. The ECE can be approximated as a weighted average of the absolute difference between the accuracy and confidence of each bin:

EQUATION

A similar metric, the maximum calibration error (MCE) (Naeini et al., 2015) , is defined as the maximum absolute difference between the confidence and accuracy of each bin:

EQUATION

A common way of visually exploring the calibration of a model is to use a reliability plot (Niculescu-Mizil and Caruana, 2005) , which plots the accuracies of the various confidence bins as a bar chart (see Figure 2 ). Reliability plots not only give a quick visual indication of a model's MCE (but not ECE, since the numbers of samples in the different bins are not shown), but also capture whether or not a model is under-confident or over-confident in general. For a perfectly calibrated model, the accuracy for each bin will match the confidence, and hence all of the bars will lie on a diagonal. By contrast, if most of the bars lie above the diagonal, meaning that the model is more accurate than it expects, then it is said to be under-confident, and if most of the bars lie below the diagonal, then it is said to be over-confident.

We also consider another metric which we call AdaECE (Adaptive ECE) where the bin sizes are decided such that each bin contains the same number of samples, unlike ECE (1) where every bin size is fixed and might contain very different number of samples (for example, Figure 2 ). However, AdaECE allows better approximation of ECE as it ensures that every approximate bin has the same sample density.

AdaECE =:



M i=1:

|Bi|

EQUATION

3 WHAT CAUSES MISCALIBRATION?

We now discuss why high-capacity neural networks, despite achieving low classification errors on well-known datasets, tend to be miscalibrated. A key empirical observation made by (Guo* et al., 2017) was that poor calibration of such networks appears to be linked to overfitting on the negative log-likelihood (NLL) during training. In this section, we further inspect this observation to provide new insights, and discuss the main factors that play a role in causing the above-mentioned NLL overfitting.

For the analysis in this section, we train a ResNet-50 network on CIFAR-10 using a training infrastructure PyTorch-CIFAR which is known to produce state-of-the-art accuracy. We train it for 350 epochs using Stochastic Gradient Descent (SGD), with a momentum of 0.9, and a learning rate of 0.1 for the first 150 epochs, 0.01 for the next 100 epochs and 0.001 for the last 100 epochs. We use a mini-batch size of 128. Following standard practice, we minimise cross-entropy loss, which, for the i th training sample, is computed as L c = -log pi,yi where pi,yi is the probability assigned by the network to the correct class for the i th sample.

It is interesting to note that NLL (a.k.a. cross-entropy) loss is a differentiable proxy for the actual metric that we aim to minimise by training a model, namely the classification error. However, there are certain differences between minimising the NLL and minimising the classification error. For instance, the NLL is minimised when for every training sample i, pi,yi = 1, whereas the classification error is minimised if for every sample i, pi,yi > pi,y for all y = y i . This indicates that even when the classification error is zero, the NLL can be positive, and the optimisation algorithm could in that case try to reduce it to zero by further increasing the value of pi,yi for each sample.

To study how miscalibration occurs during training, we plot the average NLL for the train and test sets at each training epoch in Figures 1(a ) and 1(b). We also plot the average NLL and the entropy of the softmax distribution produced by the network for the correctly and incorrectly classified samples.

In Figure 1 (c), we plot the classification errors on the train and test sets, along with the test set ECE.

Curse of misclassified samples: Figures 1(a ) and 1(b) show that although the average train NLL (for both correctly and incorrectly classified training samples) broadly decreases throughout training, after the 150 th epoch (where the learning rate drops by a factor of 10), there is a marked rise in the average test NLL, indicating that the network starts to overfit on average NLL. However, the increase in average test NLL is caused only by the incorrectly classified samples, as the average NLL for the correctly classified samples continues to decrease even after the 150 th epoch. We also observe that after epoch 150, the test set ECE rises, indicating that the network is becoming miscalibrated. This corroborates the observation in (Guo* et al., 2017) that miscalibration and NLL overfitting are linked. Peak at the wrong place: We further observe that the entropies of the softmax distributions for both the correctly and incorrectly classified test samples decrease throughout training (in other words, the distributions get peakier). This observation, coupled with the one we made above, indicates that for the wrongly classified test samples, the network gradually becomes more and more confident about its incorrect predictions. Finally, we notice that the training set classification error drops from around 20% to nearly 0% at epoch 150, and further drops to exactly 0% from epoch 250 onwards. The network thus has a very low training set classification error throughout the period of miscalibration.

Favouring correctly classified samples: One potential explanation for the above observations is that after the 150 th training epoch, the network classifies almost all of the training samples correctly, and hence, in order to further minimise the training NLL, it focuses on increasing the confidence of its correct predictions, rather than focusing on the incorrectly classified samples. In order to experimentally validate this, we use two observations. Firstly, we divide the confidence range [0, 1] into 25 bins, and present reliability plots computed on the training set at epochs 100, 200, 300 and 350 (see the top row of Figure 2 ). In Figure 2 , we also show the percentage of samples in each confidence bin. It is quite clear from these plots that over time, the network gradually pushes all of the training samples towards the highest confidence bin. Furthermore, even though the network has achieved 100% accuracy on the training set by epoch 300, it still pushes some of the samples lying in lower confidence bins to the highest confidence bin by epoch 350. It thus keeps on increasing the confidence of correct predictions even after having reached perfect accuracy on the training set. Secondly, we observe from Figure 1 (c) that between training epochs 150 and 250, i.e. between the two points where there are drops in the learning rate, the training set classification error rises slightly, before dropping to exactly 0 after epoch 250. This further confirms our hypothesis that in this period, the network focuses more on increasing the confidence of its correct predictions rather than increasing the classification accuracy on the training set. Weight magnification: The above-mentioned increase in confidence of the network's predictions can happen if the network increases its weights to increase the magnitudes of the logits. We explore this in §4. This increase in the network's confidence during training is a key cause of miscalibration.

4. IMPROVING CALIBRATION USING FOCAL LOSS:

As discussed above, overfitting on NLL, which is observed as the network grows more confident on all of its predictions irrespective of their correctness, is strongly related to poor calibration in neural networks. One cause of this phenomenon is that the cross-entropy objective minimises the difference between the softmax distribution and the ground-truth one-hot encoding over an entire mini-batch, irrespective of how well a network classifies individual samples in the mini-batch. In this work, we study an alternative loss function called focal loss (Lin et al., 2017) , which tackles this problem by weighting loss components generated from individual samples in a mini-batch by how well the model classifies them. Focal loss for the i th sample in the training set is computed as

L f = -(1 -pi,yi ) γ log pi,yi

, where γ ≥ 0 is a user-defined hyperparameter. Focal loss attributes more importance to samples that the network misclassifies than to ones that are correctly classified.

In this paper, we study properties of focal loss that help in calibration.

Empirical Observations: In order to analyse the behaviour of neural networks trained on focal loss, we use the same framework as mentioned above, and train four ResNet-50 networks on CIFAR-10, one using cross-entropy loss, and three using focal loss with γ = 1, 2 or 3. We then observe various metrics throughout the training period of these networks, for comparison. In Figures 3(a ), (b) and (c), we plot the average NLL over the test set, and for the correctly and incorrectly classified test samples, respectively, at each training epoch. In Figure 3 (d), we plot the softmax entropy for the misclassified test samples. In Figures 3(e ) and (f), we plot the average probability value pi,yi predicted for the ground-truth class, for correctly and incorrectly classified test samples, respectively. In Figure 3 (g), we plot the test set ECE for all four models against the training epoch.

Figure 3 (c) shows that in contrast to the network trained using cross-entropy, for which the NLL for misclassified test samples increases significantly after epoch 150, the rise in NLL for the networks trained on focal loss is much less severe. Moreover, from Figure 3 (d), we notice that the softmax entropy for misclassified test samples is consistently (if marginally) higher for focal loss than for cross-entropy. These observations indicate that the focal loss models are less confident about their incorrect predictions than the cross-entropy model. From Figure 3 (f), we notice that the probabilities predicted for the correct classes for misclassified test samples are slightly higher for focal loss than for cross-entropy. Conversely, Figure 3 (e) shows that the probabilities predicted for the correct classes for correctly classified test samples are somewhat lower than for cross-entropy. (Note that in Figure 3 (b), the average test NLL values for the focal loss networks on the correctly classified samples are also somewhat higher than for cross-entropy, which similarly indicates a slight decrease in confidence on those samples.) Nevertheless, for all four models, average confidence on correctly classified samples remains over 0.95 by the end of training, which is more than enough to make these predictions usable by downstream tasks. Furthermore, the classification errors on the test set for all four models are also almost the same (refer fifth row of Table 3 ). Given both of these points, along with the ECE plot in Figue 3(g), it is thus reasonable to conclude that on a practical level, the models trained on focal loss are more calibrated, whilst retaining both their classification accuracy and the majority of their confidence on correct predictions.

Theoretical Justification: As mentioned previously, once a model trained using cross-entropy reaches high accuracy on the training set, the optimisation algorithm may try to further reduce the training NLL by increasing the confidence values for the correctly classified samples, rather than focusing on the misclassified samples. One way it could achieve this would be to increase the weights of the network to increase the magnitudes of the logits. In fact, this hypothesis would help to explain the observation made in (Guo* et al., 2017 ) that models trained using some form of weight decay are better calibrated.

To verify this hypothesis, we plot the L 2 norm of the weights of the last linear layer for all four networks as a function of the training epoch (see Figure 3 (h)). It is interesting to note that although the norms of the weights for the models trained on focal loss are initially higher than that for the cross-entropy model, a complete reversal in the ordering of the weight norms occurs between epochs 150 and 250. In other words, as the network starts to become miscalibrated, the weight norm for the cross-entropy model also starts to become greater than those for the focal loss models. In practice, the reason for this is that focal loss, by design, starts to act as a regulariser on the network's weights once the model has gained a certain amount of confidence in its predictions. To better understand this, we start by considering the following proposition (see the Appendix B for a proof).

Proposition 1. For focal loss L f and cross-entropy loss L c , the gradients

∂L f ∂w = ∂Lc ∂w g(p i,yi , γ), where g(p i,yi , γ) = (1 -pi,yi ) γ -γ pi,yi (1 -pi,yi ) γ-1 log(p i,yi ), γ ≥ 0 is the focal loss hyperpa-

rameter, and w are the last layer network parameters. Thus,

∂L f ∂w ≤ ∂Lc ∂w iff g(p i,yi , γ) ∈ [0, 1].

To visualise the behaviour of g(p, γ), we plot it against p for various values of γ in Figure 4 (a). It is quite clear that g(p, γ) lies in [0, 1) for values of p that lie above a certain threshold p 0 . In the interval [0, p 0 ), however, g(p, γ) > 1. In fact, for a given threshold p 0 , we can efficiently compute (using the following proposition) a value γ * such that g(p, γ * ) < 1 for p > p 0 (see the Appendix B for a proof).

Proposition 2. For p ≥ p 0 , g(p, γ) ≤ 1 for all γ ≥ γ * = a b + 1 log a W -1 -a (1-a/b)

b log a , where a = 1 -p 0 , b = p 0 log p 0 and W -1 is the Lambert-W function (Corless et al., 1996) . Moreover, for p ≥ p 0 and γ ≥ γ * , the equality g(p, γ) = 1 holds only for p = p 0 and γ = γ * . Thus, given a p 0 , we can always use Proposition 2 to compute γ * such that (i) g(p 0 , γ * ) = 1, (ii) for p < p 0 , g(p, γ * ) > 1, and (iii) for p > p 0 , g(p, γ * ) < 1.

Implicit Regularisation: For a network trained using focal loss with a fixed γ obtained using p 0 such that g(p 0 , γ) = 1 then in the initial stages of training, when pi,yi < p 0 in general, g(p i,yi , γ) > 1 and the model gains confidence on its predictions faster than it would for cross-entropy. However, as soon as pi,yi crosses the threshold p 0 , g(p i,yi , γ) falls below 1 and reduces the size of the gradient updates made to the network weights, thereby having a regularising effect on the weights. This is why, in Figure 3 (h), we find that the weight norms of the models trained with focal loss are initially higher than that for the model trained using cross-entropy. However, as training progresses, we find that the ordering of the weight norms reverses, as focal loss starts regularising the network weights.

Figure 4(b-d ) give further insights by plotting histograms of the gradient norms of the last linear layer (over all samples in the training set) at epochs 10, 100 and 200, respectively. At epoch 10, the gradient norms for cross-entropy and focal loss are similar, but as training progresses, those for cross-entropy decrease less rapidly than those for focal loss. Finally, note from Figure 4 (a) that with higher values of γ, the fall in g(p i,yi , γ) gets steeper. We would thus expect the weight regularisation effect to be higher for models that use higher values of γ. This explains why, of the three models we trained using focal loss, the one with γ = 3 outperforms (in terms of calibration) the one with γ = 2, which in turn outperforms the model with γ = 1. Based on this observation, one might think that, in general, a higher value of gamma would lead to a more calibrated model. However, this is not the case, as we notice from Figure 4 (a) that for γ ≥ 7, g(p i,yi , γ) reduces to nearly 0 for a relatively low value of pi,yi (around 0.5). As a result, using values of γ that are too high will cause the gradients to die (i.e. reduce to nearly 0) early, at a point at which the network's predictions remain ambiguous, thereby causing the training process to fail. To this end, we provide Proposition 2 that allows us to efficiently get a sampled-dependent gamma in a principled manner. Experimentally we show that the sample-dependant γ does provide the best result among all the approaches we compare with. We also provide an empirical analysis of focal loss and cross entropy on a linear model in Appendix G.

5. EXPERIMENTS:

We perform experiments on multiple image and document classification datasets to verify the effectiveness of focal loss for training calibrated models. For image classification experiments, we use CIFAR-10 ( Krizhevsky, 2009) and CIFAR-100 (Krizhevsky, 2009) and for document classification we use 20 Newsgroups and the Stanford Sentiment Treebank (SST) (Socher et al., 2013) . We provide details on the datasets and the train/validation/test splits for each dataset in the Appendix C. On CIFAR-10 and CIFAR-100, we train the networks: ResNet-50, ResNet-110 (He et al., 2016) , Wide-ResNet-26-10 (Zagoruyko and Komodakis, 2016) , and DenseNet-121 (Huang et al., 2017) .We train the Global Pooling Convolutional Network (Lin et al., 2014) on 20 Newsgroups and the Tree-LSTM (Tai et al., 2015) on the SST Binary dataset. We provide implementation details for each of the state of the art networks in Appendix D. For each dataset-network pair, we train the network using each of the following loss functions:

Baselines (Cross-Entropy, MMCE (Kumar et al., 2018) ) and Brier Score (Brier, 1950) : Models trained on cross-entropy loss, MMCE loss (i.e. cross-entropy with an MMCE regulariser) and Brier score serve as our baselines for comparison. MMCE (Maximum Mean Calibration Error) is a continuous and differentiable proxy for calibration error and hence, can be directly optimised using standard optimisers like Stochastic Gradient Descent. Normally, MMCE is optimised as a regulariser alongside cross-entropy. Brier score, for a single sample, is computed as the squared error between the predicted softmax vector and the one-hot ground truth encoding. We find Brier score to be a particularly relevant baseline for calibration as there is a distinct penalty for increasing the confidence on incorrect classes. In addition, it is relevant as it can be decomposed into calibration and refinement (DeGroot and Fienberg, 1983) .

Focal Loss (Fixed γ): We train various models each using three values of γ set to 1, 2 and 3.

Focal Loss (Sample-Dependent γ): From Proposition 2, we know that for a given probability p, we can find a value of γ such that either g(p, γ) ∈ [0, 1), and focal loss regularises the weights of the network, or g(p, γ) > 1, and focal loss speeds up the process of making the network more confident on the correct class. In this approach, we follow the idea of accelerating the rise in the value of pi,yi (the confidence on the correct class y i for the i th training sample) as long as pi,yi < 0.5. As soon as pi,yi ≥ 0.5, we can confirm that the i th training sample has been predicted correctly. Hence, when pi,yi ≥ 0.5, we want to regularise the weights to prevent the network from becoming overconfident and miscalibrated. There can be multiple ways of choosing sample-dependent γ that satisfy the above condition. However, with just few initial tries we found these two policies to produce quite competitive results on the validation sets: (a) Focal Loss (sample-wise γ 5,3,2): γ = 5 for pi,yi ∈ [0, 0.19] (as g(0.19, 5) ≈ 1), γ = 3 for pi,yi ∈ (0.19, 0.5] and γ = 2 for pi,yi ∈ (0.5, 1], and (b) Focal Loss (sample-wise γ 5,3): γ = 5 for pi,yi ∈ [0, 0.19], and γ = 3 for pi,yi ∈ (0.19, 1].

Focal Loss (Scheduled γ): As a simplification to the above approach, we also investigated the use of a schedule for γ during training, as we expect the value of pi,yi to increase in general for all samples over time. In particular, we report results for two different schedules: (a) Focal Loss (scheduled γ 5,3,2): γ = 5 for the first 100 epochs, γ = 3 for the next 150 epochs, and γ = 2 for the last 100 epochs, and (b) Focal Loss (scheduled γ 5,3,1): γ = 5 for the first 100 epochs, γ = 3 for the next 150 epochs, and γ = 1 for the last 100 epochs. We also tried various other schedules, but found these two to produce the best results on the validation sets.

We find the best policy for each of the above three focal loss variants by cross-validating on the validation set, and report its results in Table 4 and 7 . We report results for all other policies in Appendix E.

6. DISCUSSION:

Temperature Scaling: We report the ECE % (using 15 bins), both before and after temperature scaling, for focal loss (sample-dependant which performed the best) vs baselines in Table 1 . Further, we report the same metric for different variants of focal loss in Table 4 . Similarly we report AdaECE in Table 2 and Table 5 . In fact our results show that in situations where ECE might imply that the model is well calibrated, AdaECE brings out the miscalibration present in the model. For example, in the case of WideResNet on CIFAR10 for Cross Entropy, the best ECE obtained is 0.92, implying almost no miscalibration, whereas AdaECE is 1.69 showing further scope for improvements.

We compute the optimal temperature using two different approaches: (a) learning the optimal temperature by optimising NLL over a validation set, and (b) performing grid search over temperature values between 0 and 10, with a step of 0.1, and choosing the temperature that minimises the ECE/AdaECE over a validation set. We find the second approach to produce better results in general.

Since we report ECE/AdaECE as the primary performance metric and grid search does not require a differentiable objective function, we directly minimise ECE/AdaECE over the validation set during grid search. Furthermore, since NLL is not a convex function of temperature, an optimisation algorithm may get stuck at a locally optimal temperature. We thus report the optimal temperatures and their corresponding ECEs/AdaECEs obtained using grid search (i.e. the second approach).

Performance Gains: It should be noted that for all the dataset-network pairs, we obtain state-ofthe-art classification accuracies, as shown in Table 3 . For focal loss with a fixed γ, we found that γ = 3 produced the best ECE results. This corroborates the observation we made in §4 that γ = 3 should produce better results than γ = 1 or γ = 2, as the regularising effect for γ = 3 is higher. For the sample-dependent γ approach, we found the second policy (i.e. Focal Loss (sample-wise γ 5,3) with γ = 5 for pi,yi ∈ [0, 0.19], and γ = 3 for pi,yi ∈ (0.19, 1]) to produce better results. Of the two approaches for scheduled γ, we found the first schedule (i.e. Focal Loss (scheduled γ 5,3,2) with Confident and Calibrated Models: It is interesting to note that for focal loss with sample-based γ (refer Tab. 1 and Tab. 2), beside most other focal loss models, the optimal temperatures are very close to 1, mostly lying between 0.9 and 1.1. By contrast, the optimal temperatures for the baselines (cross-entropy and MMCE) are significantly higher, with values lying between 2.0 to 2.8. An optimal temperature close to 1 indicates that the model is innately calibrated and cannot be made significantly more calibrated by temperature scaling. Furthermore, an optimal temperature that is much greater than 1 can make the network underconfident in general, as its outputs are temperature-scaled irrespective of their correctness. To see this, we follow the approach adopted in Kumar et al. (2018) , and measure the percentage of test samples that are predicted with a confidence of 0.99 or more (we call this set of test samples S99). In Table 6 , we report |S99| as a percentage of the total number of test samples, along with the accuracy of the samples in S99 for ResNet-50 and ResNet-110 trained on CIFAR-10, using cross-entropy loss, MMCE loss, and focal loss. We observe that |S99| for the focal loss model is much lower than for the cross-entropy or MMCE models before temperature scaling. However, after temperature scaling, |S99| for focal loss is significantly higher than for both MMCE and cross-entropy. The reason is that with an optimal temperature of 1.1, the confidence of the temperature-scaled model for focal loss does not reduce as much as those of the models for cross-entropy and MMCE, for which the optimal temperatures lie between 2.5 to 2.8. We thus conclude that models trained on focal loss are not only more calibrated, but also better preserve their confidence on predictions, even after being post-processed with temperature scaling. We present some qualitative results to support this claim in Appendix F.

7. CONCLUSION:

In this paper, we have shown that training using focal loss can yield multi-class classification networks that are more naturally calibrated than those trained using more conventional cross-entropy loss.

There are sound theoretical reasons to expect this: in particular, as we show in §4, focal loss implicitly regularises the weights of a network during training, reducing NLL overfitting and thereby improving calibration. Extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (20 Newsgroups/SST) datasets, and with a wide variety of different network architectures, show that this expectation is also borne out in practice. Our results show that in almost all cases, networks trained with focal loss are more calibrated than those trained with cross-entropy loss, whilst having similar levels of accuracy, making their predictions much easier for downstream components to trust. We also introduce principled way of getting sample-dependent γ for focal loss and we show that it produces good results across datasets and models.

A APPENDIX:

In § B, we show the proofs of the propositions formulated in the main text. We then describe all the datasets we used for our experiments in § C which is followed by implementation details of the state of the art networks in § D.

B PROOFS:

Here we provide the proofs of both the propositions presented in the main text. While Proposition 3 helps us understand the regularization effect of using the focal loss, the Proposition 4 efficiently provides us the γ values in a principle way such that it is sample-dependent. Implementing the sample-dependent γ is very easy as implementation of the Lambert-W function (Corless et al., 1996) is available in standard libraries (e.g. python scipy). Using it we could get quite competitive results with just few initial sample-dependent γ policies that we tried.

Proposition 3. For focal loss L f and cross-entropy loss L c , the gradients

∂L f ∂w = ∂Lc ∂w g(p i,yi , γ), where g(p i,yi , γ) = (1 -pi,yi ) γ -γ pi,yi (1 -pi,yi ) γ-1 log(p i,yi

), γ ≥ 0 is the focal loss hyperparameter, and w are the last layer network parameters. Thus,

∂L f ∂w ≤ ∂Lc ∂w iff g(p i,yi , γ) ∈ [0, 1].

Proof. Let w be the linear layer parameters connecting the feature map to the logit s. Then, using the chain rule,

∂L f ∂w = ∂s ∂w ∂ pi,y i ∂s ∂L f ∂ pi,y i . Similarly, ∂Lc ∂w = ∂s ∂w ∂ pi,y i ∂s ∂Lc ∂ pi,y i

. The derivative of the focal loss with respect to pi,yi , the softmax output of the network for the true class y i , takes the form

EQUATION

in which g(p i,yi , γ) = (1 -pi,yi ) γ -γ pi,yi (1 -pi,yi ) γ-1 log(p i,yi ) and ∂Lc ∂ pi,y i = -1 pi,y i . It is thus straightforward to verify that if g(p i,yi , γ) ∈ [0, 1], then ∂L f ∂ pi,y i ≤ ∂Lc ∂ pi,y i , which itself implies that ∂L f ∂w ≤ ∂Lc ∂w . Proposition 4. For p ≥ p 0 , g(p, γ) ≤ 1 for all γ ≥ γ * = a b + 1 log a W -1 -a (1-a/b)

b log a , where a = 1 -p 0 , b = p 0 log p 0 and W -1 is the Lambert-W function (Corless et al., 1996) . Moreover, for p ≥ p 0 and γ ≥ γ * , the equality g(p, γ) = 1 holds only for p = p 0 and γ = γ * .

Proof. We derive the value of γ > 0 for which g(p 0 , γ) = 1 for a given p 0 ∈ [0, 1]. From Proposition 4.1, we already know that

EQUATION

where L f is focal loss, L c is cross entropy loss, pi,yi is the probability assigned by the model to the ground-truth correct class for the i th sample, and

EQUATION

For p ∈ [0, 1], if we look at the function g(p, γ), then we can clearly see that g(p, γ) → 1 as p → 0, and that g(p, γ) = 0 when p = 1. To observe the behaviour of g(p, γ) for intermediate values of p, we first take its derivative with respect to p:

EQUATION

In Equation 7, γ(1 -p) γ-2 > 0 except when p = 1 (in which case γ(1 -p) γ-2 = 0). Thus, to observe the sign of the gradient ∂g(p,γ) ∂p , we focus on the term

EQUATION

Dividing Equation 8 by (-log p), the sign remains unchanged and we get

EQUATION

We can see that k(p, γ) → 1 as p → 0 and k(p, γ) → -(1 + γ) as p → 1 (using l'Hôpital's rule). Furthermore, k(p, γ) is monotonically decreasing for p ∈ [0, 1]. Thus, as the gradient ∂g(p,γ) ∂p monotonically decreases from a positive value at p = 0 to a negative value at p = 1, we can say that g(p, γ) first monotonically increases starting from 1 (as p → 0) and then monotonically decreases down to 0 (at p = 1). Thus, if for some threshold p 0 > 0 and for some γ > 0, g(p, γ) = 1, then ∀p > p 0 , g(p, γ) < 1. We now want to find a γ such that ∀p ≥ p 0 , g(p, γ) ≤ 1. First, let a = (1-p 0 ) and b = p 0 log p 0 . Then:

EQUATION

where a = (1 -p 0 ) and b = p 0 log p 0 . We know that the inverse of y = xe x is defined as x = W (y), where W is the Lambert-W function (Corless et al., 1996) . Furthermore, the r.h.s. of the inequality in Equation 10 is always negative, with a minimum possible value of -1/e that occurs at p 0 = 0.5. Therefore, applying the Lambert-W function to the r.h.s. will yield two real solutions (corresponding to a principal branch denoted by W 0 and a negative branch denoted by W -1 ). We first consider the solution corresponding to the negative branch (which is the smaller of the two solutions):

EQUATION

If we consider the principal branch, the solution is

EQUATION

which yields a negative value for γ that we discard. Thus Equation 11gives the values of γ for which if p > p 0 , then g(p, γ) < 1. In other words, g(p 0 , γ) = 1, and for any p < p 0 , g(p, γ) > 1.

C DATASET DESCRIPTION:

We use the following image-and document-classification datasets in our experiments:

1. CIFAR-10 ( Krizhevsky, 2009) : This dataset has 60,000 colour images of size 32 × 32, divided equally into 10 classes. We use a train/validation/test split of 45,000/5,000/10,000 images. 2. CIFAR-100 (Krizhevsky, 2009) : This dataset has 60,000 colour images of size 32 × 32, divided equally into 100 classes. (Note that the images in this dataset are not the same images as in CIFAR-10.) We again use a train/validation/test split of 45,000/5,000/10,000 images. 3. 20 Newsgroups: This dataset contains 20,000 news articles, categorised evenly into 20 different newsgroups based on their content. It is a popular dataset for text classification. Whilst some of the newsgroups are very related (e.g. rec.motorcycles and rec.autos), others are quite unrelated (e.g. sci.space and misc.forsale). We use a train/validation/test split of 15,098/900/3,999 documents.

4. Stanford Sentiment Treebank (SST) (Socher et al., 2013) : This dataset contains movie reviews in the form of sentence parse trees, where each node is annotated by sentiment. We use the dataset version with binary labels, for which 6,920/872/1,821 documents are used as the training/validation/test split. In the training set, each node of a parse tree is annotated as positive, neutral or negative. At test time, the evaluation is done based on the model classification at the root node, i.e. considering the whole sentence, which contains only positive or negative sentiment.

D IMPLEMENTATION DETAILS:

For training networks on the image classification datasets (CIFAR-10 and CIFAR-100), we use SGD with a momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the first 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs. We use a training batch size of 128. Furthermore, we augment the training images by applying random crops and random horizontal flips.

For 20 Newsgroups, we train the Global Pooling Convolutional Network (Lin et al., 2014) using the Adam optimiser, with learning rate 0.001, and betas 0.9 and 0.999. The code is a PyTorch adaptation of (Ng) . We used Glove word embeddings (Pennington et al., 2014) to train the network. We trained all the models for 50 epochs and used the models with the best validation accuracy.

For the SST Binary dataset, we train the Tree-LSTM (Tai et al., 2015) using the AdaGrad optimiser with a learning rate of 0.05 and a weight decay of 10 -4 , as suggested by the authors. We used the constituency model, which considers binary parse trees of the data and trains a binary tree LSTM on them. The Glove word embeddings (Pennington et al., 2014) we used were also tuned for best results.

The code framework we used is inspired by (TreeLSTM) . We trained all of our models for 25 epochs and used the models with the best validation accuracy.

For all our models, we use the PyTorch framework, setting any hyperparameters not explicitly mentioned to the default values used in the standard models. For MMCE, we used λ = 2 for all the image-classification tasks, whilst we found λ = 8 to perform better for document classification. A calibrated model which does not generalise well to an unseen test set is not very useful. Hence, for all the experiments, we set the training parameters in a way such that we get state-of-the-art test set accuracies on all datasets for each model.

E ADDITIONAL RESULTS:

We use various metrics to compare the proposed methods based on focal loss with the baselines (i.e. cross-entropy, Brier score and MMCE).

We show all test ECE of all the focal loss models before and after temperature scaling in Table 7 .

Similarly, we show all test AdaECE of all the focal loss models before and after temperature scaling in Table 8 . We present the test set error for all the focal loss models in Table 9 . We present the test NLL % before and after temperature scaling in Tables 10 and 11 , respectively. We report the test set MCE % before and after temperature scaling in Tables 12 and 13 , respectively.

In this section we use the following abbreviation to report results on different varieties of Focal Loss. FL-1 refers to Focal Loss (fixed γ 1), FL-2 refers to Focal Loss (fixed γ 2), FL-3 refers to Focal Loss (fixed γ 3), FLS-531 refers to Focal Loss (scheduled γ 5,3,1), FLS-532 refers to Focal Loss (scheduled γ 5,3,2), FLA-532 refers to Focal Loss (sample-wise γ 5,3,2) and FLA-53 refers to Focal Loss (sample-wise γ 5,3)

F QUALITATIVE RESULTS:

In Figure 8 , we present some qualitative results to show the improvement in the confidence estimates of focal loss in comparison to other baselines (i.e., cross entropy, MMCE and Brier score). For this, we take ResNet-50 networks trained on CIFAR-10 using all the four loss functions (cross entropy, MMCE, Brier score and Focal loss with sample-wise γ 5,3) and measure the confidence of their predictions for four correctly and four incorrectly classified test samples. We report these confidences both before and after temperature scaling. It is clear from Figure 8 that for all the on the misclassified samples, particularly before temperature scaling. Apart from focal loss, the only model which has relatively low confidences on misclassified test samples is the one trained using Brier score. These observations support our claim that focal loss produces not only a calibrated model but also one which is confident on its correct predictions.

G FOCAL LOSS AND CROSS ENTROPY ON A LINEAR MODEL:

To understand the effect of focal loss on simpler to analyse models, we conducted experiments on a generalised linear model on a simple data distribution.

Setup We consider a binary classification problem. The data matrix X ∈ R 2×N is created by assigning each class two normally distributed clusters such that the mean of the clusters are linearly separable. The mean of the clusters are situated on the vertices of a two-dimensional hypercube. Further to that, for 10% of the data points, the labels were flipped. The model consists of a simple 2-parameter logistic regression model. f w (x) = σ(w 1 x 1 + w 2 x 2 ). We train this model using both cross entropy and focal loss with γ = 1. Weight Magnification We have argued that focal loss implicitly regularizes the weights of the model by providing smaller gradients as compared to cross entropy. This helps in calibration as -if all the weights are large, the logits are large and thus the confidence of the network is large on all test points. When the model misclassifies, it misclassifies with a high confidence. Figure 5 shows, for a generalised linear model, that the norm of the logits and the weights of a network blows for Cross Entropy as compared to Focal Loss.

High Confidence for mistakes Figure 6 (b) and (c) shows that running gradient descent with cross entropy (CE) and focal loss (FL) both gives the same decision regions i.e. the weight vector points in the same region for both FL and CE. However, as we have seen that the norm of the weights is much larger for FL as compared to CE, we would expect the confidence of misclassified test points to be large for CE as compared to FL. Figure 6 (a) plots a histogram of the confidence of the misclassified points and it shows that CE misclassifies almost always with greater than 90% confidence whereas FL misclassifies with much lower confidence. In this section, we show that just like Cross Entropy minimizes an upper bound on the KL-Divergence between the true distribution p and the predicted distribution q, focal loss minimizes a regularized KL divergence between p and q. We use F L γ (p, q) and CE(p, q) to denote the focal loss with parameter For each image, we present the actual label, the predicted label and the confidence of the prediction both before and after temperature scaling.

:



Figures and Tables:

Figure 1: Metrics to capture how model calibration changes over training epochs in a ResNet-50 network trained on CIFAR-10.

Figure 2: Top row: reliability plots using 25 confidence bins. Bottom row: % of samples in each bin, computed over the training set at epochs 100, 200, 300 and 350.

Figure 3: Metrics to compare the calibrations of several ResNet-50 networks trained on CIFAR-10, using either cross-entropy loss, or focal loss with γ set to 1, 2 or 3.Weight magnification: The above-mentioned increase in confidence of the network's predictions can happen if the network increases its weights to increase the magnitudes of the logits. We explore this in §4. This increase in the network's confidence during training is a key cause of miscalibration.

Figure 4: (a): g(p, γ) vs p and (b-d): histogram of the gradient norms of the last linear layer for both cross entropy and focal loss.

Figure 5: (a): Norm of logits (b): Norm of weights.

Figure 6: (a): Confidence of mis-classifications (b): Decision boundary of linear classifier trained using cross entropy (c): Decision boundary of linear classifier trained using focal loss

Figure 8: Qualitative results showing the performance of Cross Entropy, Brier Score, MMCE and Focal Loss (sample-wise γ 5,3) for a ResNet-50 trained on CIFAR-10. The first row of images have been correctly classified by networks trained on all four loss functions and the second row of images have all been incorrectly classified.For each image, we present the actual label, the predicted label and the confidence of the prediction both before and after temperature scaling.

ECE (%) computed for different approaches both pre and post temperature scaling (crossvalidating T on ECE). Optimal temperature for each method is indicated in brackets.

Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="2">Cross Entropy</td><td colspan="2">Brier Loss</td><td/><td>MMCE</td><td colspan="2">Focal Loss (sample-wise γ 5,3)</td></tr><tr><td/><td/><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td></tr><tr><td/><td>ResNet 50</td><td>17.52</td><td>3.42(2.10)</td><td>6.52</td><td>3.64(1.10)</td><td>15.32</td><td>2.38(1.80)</td><td>4.50</td><td>2.00(1.10)</td></tr><tr><td>CIFAR-100</td><td>ResNet 110 Wide ResNet 26-10</td><td>19.05 15.33</td><td>4.43(2.30) 2.88(2.20)</td><td>7.88 4.31</td><td>4.65(1.20) 2.70(1.10)</td><td>19.14 13.17</td><td>3.86(2.30) 4.37(1.90)</td><td>8.56 3.03</td><td>4.12(1.20) 1.64(1.10)</td></tr><tr><td/><td>DenseNet 121</td><td>20.98</td><td>4.27(2.30)</td><td>5.17</td><td>2.29(1.10)</td><td>19.13</td><td>3.06(2.10)</td><td>3.73</td><td>1.31(1.10)</td></tr><tr><td/><td>ResNet 50</td><td>4.35</td><td>1.35(2.50)</td><td>1.82</td><td>1.08(1.10)</td><td>4.56</td><td>1.19(2.60)</td><td>1.55</td><td>0.95(1.10)</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>4.41 3.23</td><td>1.09(2.80) 0.92(2.20)</td><td>2.56 1.25</td><td>1.25(1.20) 1.25(1.00)</td><td>5.08 3.29</td><td>1.42(2.80) 0.86(2.20)</td><td>1.87 1.56</td><td>1.07(1.10) 0.84(0.90)</td></tr><tr><td/><td>DenseNet 121</td><td>4.52</td><td>1.31(2.40)</td><td>1.53</td><td>1.53(1.00)</td><td>5.10</td><td>1.61(2.50)</td><td>1.22</td><td>1.22(1.00)</td></tr><tr><td>20 Newsgroups</td><td>Global Pooling CNN</td><td>17.92</td><td>2.39(3.40)</td><td>13.58</td><td>3.22(2.30)</td><td>15.48</td><td>6.78(2.20)</td><td>6.92</td><td>2.19(1.50)</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>7.37</td><td>2.62(1.80)</td><td>9.01</td><td>2.79(2.50)</td><td>5.03</td><td>4.02(1.50)</td><td>9.19</td><td>1.83(0.70)</td></tr><tr><td>Dataset</td><td>Model</td><td colspan="2">Cross Entropy</td><td colspan="2">Brier Loss</td><td/><td>MMCE</td><td colspan="2">Focal Loss (sample-wise γ 5,3)</td></tr><tr><td/><td/><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td></tr><tr><td/><td>ResNet 50</td><td>17.52</td><td>3.67(2.10)</td><td>6.52</td><td>3.69(1.20)</td><td>15.32</td><td>2.44(1.80)</td><td>4.39</td><td>2.33(1.10)</td></tr><tr><td>CIFAR-100</td><td>ResNet 110 Wide ResNet 26-10</td><td>19.05 15.33</td><td>5.50(2.40) 2.89(2.20)</td><td>7.73 4.22</td><td>4.53(1.20) 2.81(1.10)</td><td>19.14 13.16</td><td>4.85(2.30) 4.25(1.90)</td><td>8.55 2.75</td><td>3.96(1.20) 1.63(1.10)</td></tr><tr><td/><td>DenseNet 121</td><td>20.98</td><td>5.09(2.30)</td><td>5.04</td><td>2.56(1.10)</td><td>19.13</td><td>3.07(2.10)</td><td>3.55</td><td>1.24(1.10)</td></tr><tr><td/><td>ResNet 50</td><td>4.33</td><td>2.14(2.50)</td><td>1.74</td><td>1.23(1.10)</td><td>4.55</td><td>2.06(2.50)</td><td>1.56</td><td>1.26(1.10)</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>4.40 3.23</td><td>1.84(2.70) 1.69(2.20)</td><td>2.60 1.70</td><td>1.70(1.20) 1.63(0.90)</td><td>5.06 3.29</td><td>2.45(2.70) 1.69(2.10)</td><td>2.07 1.52</td><td>1.67(1.10) 1.38(0.90)</td></tr><tr><td/><td>DenseNet 121</td><td>4.51</td><td>2.25(2.30)</td><td>2.03</td><td>2.18(0.90)</td><td>5.10</td><td>2.46(2.40)</td><td>1.42</td><td>1.42(1.00)</td></tr><tr><td>20 Newsgroups</td><td>Global Pooling CNN</td><td>17.91</td><td>2.20(3.30)</td><td>13.57</td><td>2.60(2.20)</td><td>15.21</td><td>7.07(2.10)</td><td>6.92</td><td>2.17(1.40)</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>7.31</td><td>2.10(1.80)</td><td>9.42</td><td>2.80(2.70)</td><td>4.49</td><td>3.72(1.50)</td><td>8.63</td><td>1.92(0.70)</td></tr></table>



Table content: <table/>

Error (%) computed for different approaches. In this table, FL-3 denotes Focal Loss (fixed γ 3), FLS-532 denotes Focal Loss (scheduled γ 5,3,2) and FLA-53 denotes Focal Loss (sample-wise γ 5,3) i.e. focal loss with sample-wise γ with γ = 5 for pi,yi ∈ [0, 0.19] and γ = 3 for pi,yi ∈ (0.19, 1].

Table content: <table><tr><td>Dataset</td><td>Model</td><td/><td>Cross Entropy</td><td>Brier Loss</td><td>MMCE</td><td>FL-3</td><td>FLS-532</td><td>FLA-53</td></tr><tr><td/><td>ResNet 50</td><td/><td>23.30</td><td>23.39</td><td>23.20</td><td>22.75</td><td>23.24</td><td>23.22</td></tr><tr><td>CIFAR-100</td><td colspan="2">ResNet 110 Wide ResNet 26-10</td><td>22.73 20.70</td><td>25.10 20.59</td><td>23.07 20.73</td><td>22.92 19.69</td><td>22.96 20.13</td><td>22.51 20.11</td></tr><tr><td/><td>DenseNet 121</td><td/><td>24.52</td><td>23.75</td><td>24.00</td><td>23.25</td><td>23.72</td><td>22.67</td></tr><tr><td/><td>ResNet 50</td><td/><td>4.95</td><td>5.00</td><td>4.99</td><td>5.25</td><td>5.63</td><td>4.98</td></tr><tr><td>CIFAR-10</td><td colspan="2">ResNet 110 Wide ResNet 26-10</td><td>4.89 3.86</td><td>5.48 4.08</td><td>5.40 3.91</td><td>5.08 4.13</td><td>5.71 4.46</td><td>5.42 4.01</td></tr><tr><td/><td>DenseNet 121</td><td/><td>5.00</td><td>5.11</td><td>5.41</td><td>5.33</td><td>5.65</td><td>5.46</td></tr><tr><td>20 Newsgroups</td><td colspan="2">Global Pooling CNN</td><td>26.68</td><td>27.06</td><td>27.23</td><td>29.26</td><td>28.16</td><td>27.98</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td/><td>12.85</td><td>12.85</td><td>11.86</td><td>12.19</td><td>13.07</td><td>12.80</td></tr><tr><td>Dataset</td><td>Model</td><td colspan="2">Focal Loss (fixed γ 3)</td><td colspan="2">Focal Loss (scheduled γ 5,3,2)</td><td colspan="3">Focal Loss (sample-wise γ 5,3)</td></tr><tr><td/><td/><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td colspan="2">Post T</td></tr><tr><td/><td>ResNet 50</td><td>5.13</td><td>1.97(1.10)</td><td>8.47</td><td>2.13(1.30)</td><td>4.50</td><td colspan="2">2.00(1.10)</td></tr><tr><td>CIFAR-100</td><td>ResNet 110 Wide ResNet 26-10</td><td>8.64 2.13</td><td>3.95(1.20) 2.13(1.00)</td><td>11.20 4.98</td><td>3.43(1.30) 1.94(1.20)</td><td>8.56 3.03</td><td colspan="2">4.12(1.20) 1.64(1.10)</td></tr><tr><td/><td>DenseNet 121</td><td>4.15</td><td>1.25(1.10)</td><td>7.63</td><td>1.96(1.20)</td><td>3.73</td><td colspan="2">1.31(1.10)</td></tr><tr><td/><td>ResNet 50</td><td>1.48</td><td>1.42(1.10)</td><td>2.97</td><td>1.53(1.20)</td><td>1.55</td><td colspan="2">0.95(1.10)</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>1.55 1.69</td><td>1.02(1.10) 0.97(0.90)</td><td>3.33 1.82</td><td>1.36(1.30) 1.45(1.10)</td><td>1.87 1.56</td><td colspan="2">1.07(1.10) 0.84(0.90)</td></tr><tr><td/><td>DenseNet 121</td><td>1.32</td><td>1.26(0.90)</td><td>2.22</td><td>1.34(1.10)</td><td>1.22</td><td colspan="2">1.22(1.00)</td></tr><tr><td>20 Newsgroups</td><td>Global Pooling CNN</td><td>8.67</td><td>3.51(1.50)</td><td>12.13</td><td>2.47(1.80)</td><td>6.92</td><td colspan="2">2.19(1.50)</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>16.05</td><td>1.78(0.50)</td><td>3.91</td><td>2.64(0.90)</td><td>9.19</td><td colspan="2">1.83(0.70)</td></tr></table>



Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="2">Focal Loss (fixed γ 3)</td><td colspan="2">Focal Loss (scheduled γ 5,3,2)</td><td colspan="2">Focal Loss (sample-wise γ 5,3)</td></tr><tr><td/><td/><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td></tr><tr><td/><td>ResNet 50</td><td>5.08</td><td>2.35(1.20)</td><td>8.41</td><td>2.25(1.30)</td><td>4.39</td><td>2.33(1.10)</td></tr><tr><td>CIFAR-100</td><td>ResNet 110 Wide ResNet 26-10</td><td>8.64 2.08</td><td>4.14(1.20) 2.08(1.00)</td><td>11.18 5.00</td><td>3.68(1.30) 2.11(1.20)</td><td>8.55 2.75</td><td>3.96(1.20) 1.63(1.10)</td></tr><tr><td/><td>DenseNet 121</td><td>4.15</td><td>1.23(1.10)</td><td>7.61</td><td>2.04(1.20)</td><td>3.55</td><td>1.24(1.10)</td></tr><tr><td/><td>ResNet 50</td><td>1.95</td><td>1.83(1.10)</td><td>2.95</td><td>2.18(1.20)</td><td>1.56</td><td>1.26(1.10)</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>1.62 1.84</td><td>1.44(1.10) 1.54(0.90)</td><td>3.32 2.04</td><td>1.91(1.40) 1.90(1.10)</td><td>2.07 1.52</td><td>1.67(1.10) 1.38(0.90)</td></tr><tr><td/><td>DenseNet 121</td><td>1.22</td><td>1.48(0.90)</td><td>2.19</td><td>1.59(1.20)</td><td>1.42</td><td>1.42(1.00)</td></tr><tr><td>20 Newsgroups</td><td>Global Pooling CNN</td><td>8.65</td><td>3.78(1.50)</td><td>12.13</td><td>2.18(1.90)</td><td>6.92</td><td>2.17(1.40)</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>15.64</td><td>2.17(0.50)</td><td>2.94</td><td>2.50(0.90)</td><td>8.63</td><td>1.92(0.70)</td></tr></table>

Adaptive ECE (%) computed for different approaches both pre and post temperature scaling (cross-validating T on Adaptive ECE). Optimal temperature for each method is indicated in brackets.

Table content: <table><tr><td>γ = 5 for the first 100 epochs, γ = 3 for the next 150 epochs, and γ = 2 for the last 100 epochs) to</td></tr><tr><td>produce better ECEs/AdaECEs for every model and dataset. We report the ECE values for the best</td></tr><tr><td>performing approaches in Table 1 and 4. Similarly we show the AdaECE values of these models in 2</td></tr><tr><td>and 5. Full results (ECE, AdaECE, MCE, NLL and test error) for all approaches are reported in the</td></tr><tr><td>Appendix E.</td></tr></table>

Percentage of test samples predicted with confidence higher than 99% and the corresponding accuracy for Cross Entropy, MMCE and Focal loss computed both pre and post temperature scaling (represented in the table as pre T and post T respectively).

Table content: <table/>

ECE (%) computed for different focal loss approaches both pre and post temperature scaling (cross-validating T on ECE). Optimal temperature for each method is indicated in brackets.

Table content: <table><tr><td>Dataset</td><td>Model</td><td/><td>FL-1</td><td/><td>FL-2</td><td/><td>FL-3</td><td colspan="2">FLS-531</td><td colspan="2">FLS-532</td><td colspan="2">FLA-532</td><td colspan="2">FLA-53</td></tr><tr><td/><td/><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td><td>Pre T</td><td>Post T</td></tr><tr><td/><td>ResNet 50</td><td colspan="2">12.86 2.54(1.50)</td><td>8.55</td><td>2.44(1.30)</td><td>5.08</td><td colspan="3">2.35(1.20) 11.58 2.01(1.40)</td><td>8.41</td><td>2.25(1.30)</td><td>9.08</td><td>1.94(1.30)</td><td>4.39</td><td>2.33(1.10)</td></tr><tr><td>CIFAR-100</td><td>ResNet 110 Wide ResNet 26-10</td><td colspan="4">15.08 4.30(1.60) 11.57 4.38(1.40) 8.93 2.74(1.40) 4.65 2.96(1.20)</td><td>8.64 2.08</td><td colspan="7">4.14(1.20) 14.98 3.92(1.60) 11.18 3.68(1.30) 11.74 4.21(1.30) 2.08(1.00) 9.20 2.52(1.40) 5.00 2.11(1.20) 5.00 2.58(1.20)</td><td>8.55 2.75</td><td>3.96(1.20) 1.63(1.10)</td></tr><tr><td/><td>DenseNet 121</td><td colspan="2">14.24 2.71(1.50)</td><td>7.90</td><td>2.36(1.20)</td><td>4.15</td><td colspan="3">1.23(1.10) 13.01 2.18(1.50)</td><td>7.61</td><td>2.04(1.20)</td><td>8.04</td><td>2.10(1.30)</td><td>3.55</td><td>1.24(1.10)</td></tr><tr><td/><td>ResNet 50</td><td>3.42</td><td>1.51(1.60)</td><td>2.37</td><td>1.69(1.20)</td><td>1.95</td><td>1.83(1.10)</td><td>4.06</td><td>2.36(1.50)</td><td>2.95</td><td>2.18(1.20)</td><td>2.50</td><td>1.23(1.30)</td><td>1.56</td><td>1.26(1.10)</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>3.42 2.70</td><td>1.57(1.70) 1.52(1.40)</td><td>2.69 1.64</td><td>1.29(1.30) 1.47(1.10)</td><td>1.62 1.84</td><td>1.44(1.10) 1.54(0.90)</td><td>4.91 2.75</td><td>2.62(1.60) 1.87(1.30)</td><td>3.32 2.04</td><td>1.91(1.40) 1.90(1.10)</td><td>2.78 1.68</td><td>1.58(1.30) 1.49(1.10)</td><td>2.07 1.52</td><td>1.67(1.10) 1.38(0.90)</td></tr><tr><td/><td>DenseNet 121</td><td>3.44</td><td>1.85(1.40)</td><td>1.80</td><td>1.39(1.10)</td><td>1.22</td><td>1.48(0.90)</td><td>4.11</td><td>2.20(1.50)</td><td>2.19</td><td>1.59(1.20)</td><td>2.44</td><td>1.60(1.20)</td><td>1.42</td><td>1.42(1.00)</td></tr><tr><td colspan="6">20 Newsgroups Global Pooling CNN 15.06 2.55(2.70) 12.10 3.33(1.60)</td><td>8.65</td><td colspan="7">3.78(1.50) 13.55 3.55(1.90) 12.13 2.18(1.90) 12.19 3.14(1.70)</td><td>6.92</td><td>2.17(1.40)</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>6.01</td><td>2.92(1.70)</td><td>2.67</td><td colspan="3">2.67(1.00) 15.64 2.17(0.50)</td><td>3.94</td><td>1.90(1.30)</td><td>2.94</td><td>2.50(0.90)</td><td>3.54</td><td>3.54(1.00)</td><td>8.63</td><td>1.92(0.70)</td></tr></table>

AdaECE (%) computed for different focal loss approaches both pre and post temperature scaling (cross-validating T on AdaECE). Optimal temperature for each method is indicated in brackets.

Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="7">FL-1 FL-2 FL-3 FLS-531 FLS-532 FLA-532 FLA-53</td></tr><tr><td/><td>ResNet 50</td><td colspan="3">22.80 23.15 22.75</td><td>23.49</td><td>23.24</td><td>23.55</td><td>23.22</td></tr><tr><td>CIFAR-100</td><td>ResNet 110 Wide ResNet 26-10</td><td colspan="3">22.36 22.52 22.92 19.61 20.01 19.69</td><td>22.81 20.13</td><td>22.96 20.13</td><td>22.93 19.71</td><td>22.51 20.11</td></tr><tr><td/><td>DenseNet 121</td><td colspan="3">23.82 23.19 23.25</td><td>23.69</td><td>23.72</td><td>22.41</td><td>22.67</td></tr><tr><td/><td>ResNet 50</td><td>4.93</td><td>4.98</td><td>5.25</td><td>5.66</td><td>5.63</td><td>5.24</td><td>4.98</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>4.78 4.27</td><td>5.06 4.27</td><td>5.08 4.13</td><td>6.13 4.11</td><td>5.71 4.46</td><td>5.19 4.14</td><td>5.42 4.01</td></tr><tr><td/><td>DenseNet 121</td><td>5.09</td><td>4.84</td><td>5.33</td><td>5.46</td><td>5.65</td><td>5.46</td><td>5.46</td></tr><tr><td colspan="5">20 Newsgroups Global Pooling CNN 26.13 28.23 29.26</td><td>29.16</td><td>28.16</td><td>27.26</td><td>27.98</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td colspan="3">12.63 12.30 12.19</td><td>12.36</td><td>13.07</td><td>12.30</td><td>12.80</td></tr></table>

Error (%) computed for different focal loss approaches.

Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="3">Cross Entropy Brier Loss MMCE</td><td>FL-1</td><td>FL-2</td><td>FL-3</td><td colspan="4">FLS-531 FLS-532 FLA-532 FLA-53</td></tr><tr><td/><td>ResNet 50</td><td>153.67</td><td>99.63</td><td>125.28</td><td>105.61</td><td>92.82</td><td>87.52</td><td>100.09</td><td>92.66</td><td>94.10</td><td>88.03</td></tr><tr><td>CIFAR-100</td><td>ResNet 110 Wide ResNet 26-10</td><td>179.21 140.10</td><td>110.72 84.62</td><td>180.54 119.58</td><td>114.18 87.56</td><td>96.74 77.80</td><td>90.90 74.66</td><td>112.46 88.61</td><td>95.85 78.52</td><td>97.97 78.86</td><td>89.92 76.92</td></tr><tr><td/><td>DenseNet 121</td><td>205.61</td><td>98.31</td><td>166.65</td><td>115.50</td><td>93.11</td><td>87.13</td><td>107.91</td><td>93.12</td><td>91.14</td><td>85.47</td></tr><tr><td/><td>ResNet 50</td><td>41.21</td><td>18.67</td><td>44.83</td><td>22.67</td><td>18.60</td><td>18.43</td><td>25.32</td><td>20.50</td><td>18.69</td><td>17.55</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>47.51 26.75</td><td>20.44 15.85</td><td>55.71 28.47</td><td>22.54 17.66</td><td>19.19 14.96</td><td>17.80 15.20</td><td>32.77 18.50</td><td>22.48 15.57</td><td>19.39 14.78</td><td>18.54 14.55</td></tr><tr><td/><td>DenseNet 121</td><td>42.93</td><td>19.11</td><td>52.14</td><td>22.50</td><td>17.56</td><td>18.02</td><td>27.41</td><td>19.50</td><td>20.14</td><td>18.39</td></tr><tr><td colspan="2">20 Newsgroups Global Pooling CNN</td><td>176.57</td><td>130.41</td><td>158.70</td><td colspan="3">140.40 115.97 109.62</td><td>128.75</td><td>123.72</td><td>124.03</td><td>109.17</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>50.20</td><td>54.96</td><td>37.28</td><td>53.90</td><td>47.72</td><td>50.29</td><td>50.25</td><td>53.13</td><td>45.08</td><td>49.23</td></tr></table>

NLL (%) computed for different approaches post temperature scaling (cross validated on ECE).

Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="8">Cross Entropy Brier Loss MMCE FL-1 FL-2 FL-3 FLS-531 FLS-532 FLA-532 FLA-53</td></tr><tr><td/><td>ResNet 50</td><td>44.34</td><td>36.75</td><td>39.53</td><td>33.22 21.03 13.02</td><td>26.76</td><td>23.56</td><td>22.40</td><td>16.12</td></tr><tr><td/><td>ResNet 110</td><td>55.92</td><td>24.85</td><td>50.69</td><td>40.49 32.57 26.00</td><td>37.24</td><td>29.56</td><td>34.73</td><td>22.57</td></tr><tr><td>CIFAR-100</td><td>Wide ResNet 26-10 DenseNet 121</td><td>49.36 56.28</td><td>14.68 15.47</td><td>40.13 49.97</td><td>27.00 15.14 35.45 21.70 11.61 9.96</td><td>27.81 38.68</td><td>17.59 18.91</td><td>13.64 21.34</td><td>10.17 9.68</td></tr><tr><td/><td>ResNet 50</td><td>38.65</td><td>31.54</td><td>60.06</td><td>31.75 25.00 21.83</td><td>30.54</td><td>23.57</td><td>25.45</td><td>14.89</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>44.25 48.17</td><td>25.18 77.15</td><td>67.52 36.82</td><td>73.35 25.92 25.15 29.17 30.17 23.86</td><td>34.18 37.57</td><td>30.38 30.65</td><td>30.80 18.51</td><td>18.95 74.07</td></tr><tr><td/><td>DenseNet 121</td><td>45.19</td><td>19.39</td><td>43.92</td><td>38.03 29.59 77.08</td><td>33.50</td><td>16.47</td><td>17.85</td><td>13.36</td></tr><tr><td colspan="2">20 Newsgroups Global Pooling CNN</td><td>36.91</td><td>31.35</td><td>34.72</td><td>34.28 24.10 18.85</td><td>26.02</td><td>25.02</td><td>24.29</td><td>17.44</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>71.08</td><td>92.62</td><td>68.43</td><td>95.48 86.21 22.32</td><td>76.28</td><td>86.93</td><td>80.85</td><td>73.70</td></tr></table>

MCE (%) computed for different approaches pre temperature scaling.

Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="10">Cross Entropy Brier Loss MMCE FL-1 FL-2 FL-3 FLS-531 FLS-532 FLA-532 FLA-53</td></tr><tr><td/><td>ResNet 50</td><td>12.75</td><td>21.61</td><td>11.99</td><td>8.92</td><td>8.86</td><td>6.76</td><td>7.46</td><td>6.76</td><td>5.24</td><td>27.18</td></tr><tr><td/><td>ResNet 110</td><td>22.65</td><td>13.56</td><td>19.23</td><td colspan="3">20.13 12.00 13.06</td><td>18.28</td><td>13.72</td><td>15.89</td><td>10.94</td></tr><tr><td>CIFAR-100</td><td>Wide ResNet 26-10 DenseNet 121</td><td>14.18 21.63</td><td>13.42 8.55</td><td>16.50 13.02</td><td colspan="2">10.28 18.32 10.49 11.63</td><td>9.96 6.17</td><td>13.18 6.21</td><td>11.01 6.48</td><td>12.50 9.41</td><td>9.73 5.68</td></tr><tr><td/><td>ResNet 50</td><td>20.60</td><td>22.46</td><td>23.60</td><td colspan="3">25.86 28.17 15.76</td><td>22.05</td><td>23.85</td><td>24.76</td><td>26.37</td></tr><tr><td>CIFAR-10</td><td>ResNet 110 Wide ResNet 26-10</td><td>29.98 26.63</td><td>22.73 77.15</td><td>31.87 32.33</td><td colspan="3">29.74 23.82 37.61 74.58 29.58 25.64</td><td>26.25 28.63</td><td>25.94 20.23</td><td>11.59 19.68</td><td>17.35 36.56</td></tr><tr><td/><td>DenseNet 121</td><td>32.52</td><td>19.39</td><td>27.03</td><td colspan="3">19.68 22.71 76.27</td><td>21.05</td><td>32.76</td><td>35.06</td><td>13.36</td></tr><tr><td colspan="2">20 Newsgroups Global Pooling CNN</td><td>6.10</td><td>13.04</td><td>15.12</td><td>7.79</td><td colspan="2">11.88 22.22</td><td>16.86</td><td>11.85</td><td>8.37</td><td>12.65</td></tr><tr><td>SST Binary</td><td>Tree LSTM</td><td>88.48</td><td>91.86</td><td>32.92</td><td colspan="3">87.77 86.21 74.52</td><td>54.27</td><td>88.85</td><td>82.42</td><td>76.71</td></tr></table>

MCE (%) computed for different approaches post temperature scaling (cross validated on ECE).

Table content: <table/>

Peer review (iclr2020):
The paper describes how the use of the now-standard focal loss can lead to improved calibration results when used to fit deep-models. When fitting a large capacity model with NLL, the model can often try to drive its predictions close to 1 (i.e. infinity pre-softmax) on the training set, ultimately leading to poorly calibrated models and overfitting behaviour. The focal loss appears to mitigate this issue. The approach is extremely simple to implement, the theoretical justifications are believable, and the calibration/accuracy performances seem to be good -- for this reasons, I think that the paper should be accepted. (1) it would be interesting to compare the approach to using the standard cross-entropy applied to smoothed labels (i.e. (1-eps,eps) instead of (1,0) in binary classification and obvious generalisation in multi-class setting). (2) data-augmentation often greatly helps with calibration -- the paper did not describe in details what has been done on that front for the numerical investigations.