Year: 2021

Abstract:
Much recent work has been devoted to the problem of ensuring that a neural network’s confidence scores match the true probability of being correct, i.e. the calibration problem. Of note, it was found that training with focal loss leads to better calibrated deep networks than cross-entropy, while achieving the same level of accuracy Mukhoti et al. (2020). This success stems from focal loss regularizing the entropy of the network’s prediction (controlled by the hyper-parameter γ), thereby reining in the network’s overconfidence. Further improvement is expected if γ is selected independently for each training sample. However, the proposed Sample-Dependent Focal Loss (FLSD) in Mukhoti et al. (2020) is based on simple heuristics that does not take into account the difference in the network’s calibration behaviour for different samples (or groups of samples). As a result it is only slightly better than focal loss with fixed γ. In this paper, we propose a calibration-aware version of FLSD, called AdaFocal, which, at every training step t, adaptively modifies the γ for individual group of samples based on (1) γt−1 from the previous training step (2) the magnitude of the network’s under/over-confidence for those groups. We evaluate our method on various small to large-scale image recognition tasks and one NLP task, covering a variety of network architectures, to confirm that AdaFocal consistently achieves improved calibration without a significant loss in accuracy. Further, the models trained with AdaFocal are shown to have significantly improved Out-of-Distribution (OOD) detection capability.

heading: 1 INTRODUCTION
text: Neural networks have found tremendous success in almost every field including computer vision, natural language processing, and speech recognition. Over time, these networks have grown complex and larger in size to achieve state-of-the-art performance and they continue to evolve further in that direction. However, it has been well established that such high capacity networks suffer from poor calibration Guo et al. (2017), i.e. the confidence scores of the predictions do not reflect the real world probabilities of those predictions being true. For example, if the network assigns 0.8 confidence to a set of predictions, we should expect 80% of those predictions to be correct. However, this is far from reality since modern networks tend to be grossly over-confident. This is of great concern, particularly for mission-critical applications such as autonomous driving, medical diagnosis, wherein the downstream decision making not only rely on the predictions but also on their confidence.
In recent years, there has been a growing interest in developing methods for calibrating neural networks. These can be mainly divided into two categories (1) post-hoc approaches that perform calibration after training (2) methods that calibrate the model during training itself. The first includes methods such as Platt scaling Platt (1999), histogram binning Zadrozny & Elkan (2001), Isotonic regression Zadrozny & Elkan (2002), Bayesian binning and averaging Naeini et al. (2015); Naeini & Cooper (2016), and Spline fitting Gupta et al. (2021). Methods in the second category focus on training the model on an objective function that accounts for calibration as well, including Maximum Mean Calibration Error (MMCE) Kumar et al. (2018), Label smoothing Müller et al. (2019), and recently focal loss Mukhoti et al. (2020). These methods aim to produce inherently calibrated models which when combined with post training calibration methods lead to further improvements.
Contribution. Our work falls into the second category. We build upon the calibration properties of focal loss to propose a modification that further improves its performance. Firstly, we make the observation that while regular focal loss, with a fixed γ parameter, improves the overall calibration by preventing samples from being over-confident, it also leaves other samples under-confident. To
address this drawback, we propose a modification to the focal loss called AdaFocal that adjusts the γ for each training sample (or rather a group of samples) separately by taking into account the model’s under/over-confidence about a similar corresponding group in the validation set. We evaluate the performance of our method on four image classification tasks: CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, and one text classification task: 20 Newsgroup, using various model architectures, and show that AdaFocal substantially outperforms the regular focal loss and other state-of-the-art calibration techniques in the literature. We further study the performance of AdaFocal on an out-of-distribution detection task and find it to perform better than the competing methods. Finally, we find that the models trained using AdaFocal get innately calibrated to a level that most times do not significantly benefit from temperature scaling.

heading: 2 PROBLEM SETUP AND DEFINITIONS
text: Consider a classification setting where we are given a set of training data {(xn, ytrue,n)}, with xn ∈ X being the input and ytrue,i ∈ Y = {1, 2, . . . ,K} the associated ground-truth label. Using this data we wish to train a classifier fθ(x) that outputs a vector p̂ over theK classes. We also assume access to a validation set for hyper-parameter tuning and a test set for evaluating its performance. For example, fθ(·) can be a neural network with learnable parameters θ, x is an image, and p̂ is the output of a softmax layer whose kth element p̂k is the probability score for class k. We refer to ŷ = argmaxk∈Y p̂k as the network’s prediction and the associated probability score p̂ŷ as the predicted confidence, and the same quantity for the jth example is p̂ŷ,j .
In this setting, a network is said to be perfectly calibrated if the predicted confidence p̂ŷ reflects the true probability of the network classifying x correctly i.e. P(ŷ = ytrue | p̂ŷ = p) = p, ∀p ∈ [0, 1] Guo et al. (2017). Continuing our example, if the network assigns an average confidence score of 0.8 to a set of predictions then we should expect 80% of those to be correct. We define Calibration Error as E = p̂ŷ − P(ŷ = ytrue | p̂ŷ) and the Expected Calibration Error as Ep̂ŷ [E ] = Ep̂ŷ [ |p̂ŷ − P(ŷ = ytrue | p̂ŷ)| ] Guo et al. (2017). However, as the true calibration error cannot be computed empirically with a finite sized dataset, the following three approximations are generally used in the literature. That is, for a dataset {(xn, ytrue,n)}Nn=1, (1) ECE = ∑M i=1 |Bi| N |Ci − Ai| Guo et al. (2017), where Bi is equal-width bin that contains all examples j with p̂ŷ,j in the range [ iM , i+1 M ), Ci = 1 |Bi| ∑ j∈Bi p̂ŷ,j is the average confidence and Ai = 1 |Bi| ∑ j∈Bi 1(ŷj = ytrue,j) is the bin accuracy. Note that Ei = Ci − Ai is the empirical approximation of the calibration error E , (2) AdaECE = ∑M i=1 |Bi| N |Ci − Ai| Nguyen & O’Connor (2015), where ∀i, j |Bi| = |Bj | are adaptively sized (equal-mass) bins that contain an equal number of samples, and (3) ClasswiseECE Kumar et al. (2018); Kull et al. (2019) estimates the calibration over all K classes: ClasswiseECE = 1 K ∑M i=1 ∑K k=1 |Bi,k| N |Ci,k −Ai,k| where Ci,k = 1 |Bi,k| ∑ j∈Bi,k p̂k,j is the average confidence for
the kth class and Ai,k = 1|Bi,k| ∑ j∈Bi,k 1(ytrue,j = k) is the accuracy of the kth class in the ith bin.
Lastly, as ECE has been shown to be a biased estimate of true calibration Vaicenavicius et al. (2019), we additionally use two de-biased estimates of ECE namely ECEdebiased proposed in Kumar et al. (2019) and ECEsweep proposed in Roelofs et al. (2021) to further confirm our results.

heading: 3 CALIBRATION PROPERTIES OF FOCAL LOSS
text: Focal loss Lin et al. (2017) LFL(p) = −(1 − p)γ log p was originally proposed to improve the accuracy of classifiers by focusing on hard examples and down-weighting well classified examples. Recently it was further shown that focal loss may also result in significantly better calibrated models than cross entropy Mukhoti et al. (2020). This is because, based on the relation: LFL ≥ KL(q||p̂)− γH(p̂) where q is the one-hot target vector, focal loss while minimising the main KL divergence objective also increases the entropy of the prediction p̂. As a consequence this prevents the network from being overly confident on wrong predictions and overall improves calibration.
The regular focal loss with fixed γ, as we show in this section, does not achieve the best calibration. In Figure 1, we plot the calibration behaviour of ResNet50 in different bins when trained on CIFAR-10 with different focal losses. The ith bin’s calibration error subscripted by "val"Eval,i = Cval,i−Aval,i is computed on the validation set using 15 equal-mass binning. The figure shows the lowest (bin-0), a middle (bin-7) and highest bin (bin-7). For reference, the rest of the bins and their bin boundaries are
shown in Appendix B. From Figure 1 (a), we see that although focal loss γ = 4 achieves the overall lowest calibration error (AdaECE), there’s no single γ that performs the best across all the bins. For example, in bin-0 γ = 4, 5 seems to achieve better calibration whereas γ = 0, 3 are over-confident. For bin-7, on the other hand, γ = 3 seems to be better calibrated whereas γ = 4, 5 are under-confident and γ = 0 is over-confident.
This clearly indicates that using different γs for different bins can further improve the calibration. Such an attempt is presented in Mukhoti et al. (2020) called the Sample-Dependent Focal Loss (FLSD-53) which assigns γ = 5 if the training sample’s true class posterior p̂ytrue ∈ [0, 0.2) and γ = 3 if p̂ytrue ∈ [0.2, 1]. However, this strategy is fixed for every dataset-model pair and is based on simple heuristics of choosing higher γ for smaller values of p̂ytrue and relatively lower γ for higher values of p̂ytrue . However, from Figure 1(b), we see that FLSD-53 is also not the best strategy across all the bins. This, therefore, motivates the design of a γ selection strategy that can assign an appropriate γ for each bin based on the magnitude and sign of Eval,i. However, in order to design such a strategy we need solutions to the following two major challenges:
1. How do we find some correspondence between the "confidence of training samples", which we can manipulate during training by adjusting the entropy regularising parameter γ, and the "confidence of the validation samples", which we want to be actually manipulated but do not have direct control over? In other words, in order to indirectly control the confidence of a particular group of validation samples, how do we know which particular group of training samples’ confidence to be manipulated?
2. Given that there is a correspondence between a training group and a validation group (even if it’s loose), how do we arrive at the exact values of γ that will lead to better calibration?
We try to answer the first question in the next section and the answer to the second question leads to AdaFocal which is the main contribution of the paper.

heading: 4 CORRESPONDENCE BETWEEN CONFIDENCE OF TRAIN AND VAL. SAMPLES
text: In order to find some correspondence, an intuitive thing to do would be to group the validation samples into M equal-mass validation-bins, and then use these validation-bin boundaries to group the training samples as well. Then, we can compare the average confidence of the validation samples and the average confidence of the training samples, in the same validation-bin, to check for any correspondence.
Quantities of interest For binning validation samples, we always look at the confidence of the top predicted class ŷ denoted by p̂val,top (bin average: Cval,top). For training samples, on the other hand, instead of the confidence of the top predicted class ŷ denoted by p̂train,top (bin average: Ctrain,top), we will focus on the confidence of the true class ytrue denoted by p̂train,true (average: Ctrain,true) because during training we only care about p̂train,true which is manipulated through some loss function. For reference however, Figure 10 in Appendix C compares Ctrain,true and Ctrain,top to show that as the training set accuracy approaches 100%, the top predicted class and the true class for a training sample become the same. Henceforth, for a cleaner notation, we will always refer to Ctrain ≡ Ctrain,true and Cval ≡ Cval,top.
Common binning When training samples are grouped using the bin boundaries of the validationbins. In Figure 2(b), we compareCtrain,i in validation-bin-i 1 withCval,i in the same validation-bin-i, and find that there is indeed a good correspondence between the two quantities. For example in Figure 2(b), as γ increases from 0, 3 to 5, the solid-line (Ctrain,i) gets lower, and the same behaviour is observed on the starred-line (Cval,i) as well. For completeness, rest of the bins are shown in Figure 12 Appendix C. This is very encouraging as now we can expect (even though loosely) that if we increase/decrease the confidence of a group of training samples in some lower (or middle, or higher) probability region then the same will be reflected on a similar group of validation samples in lower (or middle, or higher) probability region. This therefore provides a way to indirectly control the value of Cval,i by manipulating Ctrain,i, and from a calibration point of view, our strategy going forward would be to exploit this correspondence to keep Ctrain,i (which we have control over during training) closer to Aval,i (the validation set accuracy in validation-bin-i) so that Cval,i also stays closer to Aval,i to overall reduce the calibration error Eval,i = Cval,i −Aval,i.
Independent binning Before proceeding, for completeness, we also look at the case when training samples and validation samples are grouped independently into their respective training-bins and validation-bins. Figure 2(a) compares Ctrain,i in training-bin-i with Cval,i in validation-bin-i. We observe a similar behaviour as mentioned above. Note that since the binning is independent, the boundaries of training-bin-imay not be exactly the same as that of validation-bin-i, however as shown in Figure 11 Appendix C (along with rest of the bins and their bin boundaries), they are quite close, meaning that a training group in lower (/middle/higher) probability region have good correspondence with the validation group in a similar nearby region.
Going forward, for the ease of algorithm design, we will simply stick to the case of "common binning" where training samples are grouped as per validation-bin boundaries. This will allows us to maintain a one-to-one correspondence between the boundaries of the ith training and validation group.

heading: 5 PROPOSED METHOD
text: Let’s denote the nth training sample’s true class posterior p̂ytrue by pn. Given that pn falls into validation-bin-b, our goal is to keep pn, or as per the discussion above its averaged equivalent Ctrain,b, closer to Aval,b so that the same is reflected on Cval,b. For manipulating pn, we will utilize the regularization effect that focal loss’s parameter γ has on the confidence of the predictions Mukhoti
1It may happen that no training sample belong to a particular validation-bin’s boundaries. In that case, Ctrain,i has been shown to drop to zero for example in bin-14 in Figure 2 (b).
et al. (2020). At this point, one can choose to update γb either based on (1) how far pn is from Aval,b i.e. γ = f(pn − Aval,b) or (2) how far Cval,b is from Aval,b i.e. γ = f(Cval,b − Aval,b). Such a γ-update-rule should ensure that whenever the model is over-confident, i.e. pn > Aval,b (or Cval,b > Aval,b), γ is increased so that the gradients get smaller which prevents pn from increasing further. On the other hand, when pn < Aval,b (or Cval,b < Aval,b), i.e. the model is under-confident, we decrease γ so as to get larger gradients that in turn will increase pn 2.
Based on this discussion, next we design and study a calibration-aware γ-update strategy called CalFocal, which with some additional modifications lead to AdaFocal.

heading: 5.1 CALIBRATION AWARE FOCAL LOSS (CALFOCAL)
text: Case 1: γ = f(pn −Aval,b) Treating Aval,b as the point that we want pn to not deviate from, we make the focal loss parameter γ a function of pn −Aval,b to get
LCalFocal(pn) = −(1− pn)γn log pn, with, γn = exp(λ(pn −Aval,b)), (1)
where, b is the validation-bin in which pn falls. The hyper-parameter λ is the scaling factor which combined with the exponential function helps to quickly ramp up/down γ. The exponential function adheres to the γ-update rule mentioned earlier and also ensures γ is > 0. Figure 3(a) plots LCalFocal vs. pn for Aval,b = 0.8. We see that based on the strength of λ, the loss drastically drops near pn = 0.8 and thereafter remains close to zero. This shows that LCalFocal aims is to first push p towards 0.8 and then slow its growth towards overconfidence. Next, in Figure 3(c), we find that CalFocal with λ = 10, 100 is able to reduce the calibration error compared to cross entropy but it is still far from FLSD-53’s performance. Also note in Figure 3(b) that too high λ (=100) affects the accuracy of model. Most importantly, Figure 3(d) compares Ctrain,i with Cval,i (and also Aval,i) for bin-0, where we find some evidence that the strategy of bringing pn or Ctrain,i (solid lines) closer to Aval,i (dashed lines) results in Cval,i (starred lines) getting closer to Aval,i as well, thus reducing the calibration error Eval,i = Cval,i −Aval,i slightly.
Case 2: γ = f(Cval,b − Aval,b) Note that Eq. 1 assigns a different γn for each training sample. To reduce computation and avoid using a different γn for each training sample, one can instead use a common γb for all the training samples that fall into the validation-bin-b by simply making it a function of Cval,b −Aval,b instead of pn −Aval,b.
LCalFocal(pn) = −(1− pn)γb log pn, with, γb = exp(λ(Cval,b −Aval,b)) (2)
where, b is the validation-bin in which pn falls. As shown in Appendix D, it’s performance is very similar (or slightly better than) CalFocal in Eq. 1. Further, it makes more sense to update γ based on how far Cval,b is from Aval,b instead of how far pn is from Aval,b because, as shown in Figure 3(d) bin-0, one may find Cval,b (starred lines) quite closer to Aval,b (dashed lines) even when pn or its
2Note that for focal loss increasing γ does not always lead to smaller gradients. This mostly holds true in the region pn approximately > 0.2 (see Figure 3(a) in Mukhoti et al. (2020)). However, in practice and as shown by the training-bin boundaries of bin-0 and bin-1 in Figure in Figure 11 Appendix C, we find majority of the training samples to lie above 0.2 during the majority of the training, and therefore, for the experiments in this paper, we simply stick to the rule of increasing γ to decrease gradients and stop pn from increasing and vice versa.
average equivalent Ctrain (solid lines) is far from Aval,b. At this point when Cval,b = Aval,b, we should stop updating γ further, even though pn −Aval,b 6= 0, as we have reached our goal of making Eval,b = Cval,b −Aval,b = 0. Therefore, we use Eq. 2 of Case 2 as base for AdaFocal.
Limitations of CalFocal: (1) Let’s say at some point of training, a high γb over the next few epochs reduces the calibration error Eval,b = Cval,b −Aval,b. Then, it is desirable to continue the training with the same high γb. However, note CalFocal’s update rule in Eq. 2 which will reduce γ → 1 as the Cval,b − Aval,b → 0. (2) At some point let’s say Cval,b − Aval,b is quite high. This will set γb to some high value as well depending on the hyper-parameter λ. Assuming this γb is still not high enough to bring down the confidence, we would want a way to further increase γb. However, CalFocal is incapable of doing so as it will continue to hold at γb = exp(λ(Cval,b − Aval,b)). By addressing these two issues in the next sub-section we present the final algorithm for AdaFocal.

heading: 5.2 CALIBRATION-AWARE ADAPTIVE FOCAL LOSS (ADAFOCAL)
text: A straightforward way to address the above limitations is to make γb,t depend on γb,t−1 i.e.
L(pn, t) = −(1− pn)γb,t log pn, with, γb,t = γb,t−1 ∗ exp(Cval,b −Aval,b). (3) This update rule address the limitations of CalFocal in the following way. Let’s say at some point we observe over-confidence i.e. Eval,b = Cval,b − Aval,b > 0. Then, in the next step γb will be increased. In the subsequent steps, it will continue to increase unless the calibration error Eval,b starts decreasing (this additional increase in γ was not possible with CalFocal). At this point, if we find Eval,b to start decreasing, that would reduce the increase in γb over the next epochs and γb will ultimately settle down to a value when Eval,b = 0 (CalFocal at Eval,b = 0 will cause γ to go down to 1). Next, if this current value of γb starts causing under-confidence i.e. Cval,b −Aval,b < 0, then the update rule will kick in to reduce γ thus allowing Cval,b to be increased back to Aval,b. This oscillating behaviour of AdaFocal around the desired point of Cval,b = Aval,b is its main adavantage in reducing calibration error in every bin. Additionally, also note the absence of the hyper-parameter λ in the exponent of Eq. 3 which makes AdaFocal hyper-parameter free.
Finally, note an undesirable property of Eq. 3 which is the unbounded exponential update. This may easily cause γt to explode as it can be expanded as γt = γt−1 exp(Eval,t) = γ0 exp(Eval,0 + Eval,1 + ... + Eval,t−1 + Eval,t). Thus if Eval,t > 0 for quite a few number of epochs, γt will become so large that even if Eval,t < 0 in the subsequent epochs, it may not decrease to a desired level. We remedy this by simply constraining γt to an upper bound γmax to get the AdaFocal loss as
LAdaFocal(pn, t) = −(1− pn)γb,t log pn, with, γb,t = min{γmax, γb,t−1 ∗ eCval,b−Aval,b} (4) An algorithmic description of training with AdaFocal (or CalFocal) is given in Algorithm 1. Limitation: One may argue that γmax is again a hyper-parameter; however, note that it does not require any special fine-tuning. Its sole purpose is to stop γ from exploding and any reasonable value around 20 works quite well in practice. For all our experiments, we use γmax = 20. For comparison of AdaFocal with γmax = 20, γmax = 50 and unconstrained γmax =∞, please refer to Appendix L.

heading: 6 EXPERIMENTS
text: Experimental setup We evaluate the performance of our proposed method on image and text classification tasks. For image classification, we use CIFAR-10, CIFAR-100 Krizhevsky (2009), Tiny-ImageNet Deng et al. (2009), and ImageNet Russakovsky et al. (2015) to analyze the calibration of ResNet50, ResNet-100 He et al. (2016), Wide-ResNet-26-10 Zagoruyko & Komodakis (2016), and DenseNet-121 Huang et al. (2017) models. For text classification, we use the 20 Newsgroup dataset Lang (1995) and train the Global Pooling CNN model Lin et al. (2014). Further details about the datasets, models and experimental configurations are given in Appendix E.
Baseline As baseline calibration methods we use MMCE Kumar et al. (2018), Brier loss Brier (1950), Label smoothing Müller et al. (2019) and sample-dependent focal loss FLSD-53. We also report the effect of temperature scaling Guo et al. (2017) on top of these calibration methods. Following Mukhoti et al. (2020), we select the optimal temperature that produces the minimum ECE on the validation set by searching in the interval (0, 10] with step size of 0.1.
Results. In Figure 4, we compare AdaFocal against cross entropy (CE) and FLSD-53 for ResNet-50 trained on various small to large-scale image datasets. We chose FLSD-53 as our competitive baseline
as it was shown to be consistently better than MMCE, Brier Loss and Label smoothing Mukhoti et al. (2020) across many datasets-model pairs. The figure plots the test set error and ECE calibration metric. In Figure 5, for ResNet-50 on CIFAR-10 and ImageNet, we plot (1) the calibration statistics Eval = Cval − Aval of the validation set and (2) the dynamics of associated γt used by AdaFocal during the training for a few bins covering lower, middle, and higher probability regions.
From these figures, we first observe that for CIFAR-10, CIFAR-100 and Tiny-ImageNet, FLDS-53 is much better calibrated than CE. This is because, as shown in Figure 5(a) for ResNet-50 and CIFAR-10, CE is over-confident compared to FLSD-53 in every bin. For ImageNet, however, the behaviour is reversed: FLSD-53 is poorly calibrated than CE. The reason, as shown in Figure 5(b), is that due to the use of high values γ = 5, 3, FLSD-53 makes the model largely under-confident in each bin, leading to an overall high calibration error. This shows that FLSD-53 is a strategy based on heuristic (from a limited number of dataset-model pairs) that does not generalize well. AdaFocal, on the other hand, is well calibrated for all the four dataset-model pairs while achieving similar accuracy.
The dynamics/evolution of γt during training for different bins is shown in Figure 5: (1) for CIFAR10, we find γt to be closer to 1 for higher bins and closer to 20 for lower bin. These γs found by AdaFocal result in better calibration than γ = 5, 3 of FLSD-53. (2) for ImageNet, we find AdaFocal’s
γ → 0. This makes sense because for ImageNet, from Figure 4(d), cross entropy (i.e. γ = 0 for every bin) is much better calibrated than FLSD-53 and AdaFocal (starting from γ = 1) also ultimately settles down to CE (γ = 0) to achieve a similar level of calibration. This confirms that during training, unlike CE or FLSD-53, AdaFocal being aware of the network’s current under/over-confidence is able to guide the γs to the values that maintain a well calibrated model at every step. Also note that for an unseen dataset-model pair there’s no way to know beforehand which γ will perform better but these empirical evidence show that AdaFocal will automatically find those appropriate γs.
Rest of the experiments are shown in Table 1 (ECE) and Table 2 (Error)3. From Table 1, we observe that prior to temperature scaling AdaFocal outperforms the baseline methods by a substantial margin in 9 out of 11 cases. With post-temperature scaling included, AdaFocal achieves the lowest calibration error in 7 out of the 11 experiments. Further, observe that in many cases temperature scaling on top of AdaFocal does not offer any improvement (optimal temperature = 1). For the rest, the optimal temperature is close to 1 indicating that AdaFocal produces innately calibrated models during training itself. The consistency of AdaFocal across other calibration metrics is shown through AdaECE and classwise-ECE in Appendix F. ECEdebias (15 and 30 bins), ECEEW−sweep (equal-width), and ECEEM−sweep (equal-mass) are reported in Appendix G. Significance of the results is confirmed through ECE error bars with mean and standard deviations computed over 5 runs in Appendix H.
Number of bins The ECE metrics in the paper are reported using 15 bins. For AdaFocal training we experiment with 5, 10, 15, 20, 30, and 50 equal-mass (adaptive) binning when drawing calibration statistics form the validation set as reported in Appendix I. We find the best results to be from the range 10 to 20. Performance degrades when the number of bins are too small (< 10) or too large (> 20), therefore, for the AdaFocal training in the paper we use 15 bins as well.
Out-of-Distribution (OOD) detection. Following Mukhoti et al. (2020), we report the performance of AdaFocal on an OOD detection task. We train ResNet-110 and Wide-ResNet26-10 on
3While reproducing the baseline experiments in Mukhoti et al. (2020) we obtained very similar results, therefore, we simply borrow the exact values to maintain consistent comparison.

heading: Dataset Model Cross Entropy Brier Loss MMCE LS-0.05 FLSD-53 AdaFocal
text: CIFAR-10 as the in-distribution data and test on SVHN Netzer et al. (2011) and CIFAR-10-C Hendrycks & Dietterich (2019) (with level 5 Gaussian noise corruption) as OOD data. Using entropy of the softmax as the measure of uncertainty, the corresponding ROC plots are shown in Figure 6 and AUROC scores are reported in Table 10 in Appendix J. We see that models trained with AdaFocal outperform focal loss γ = 3 (FL-3) and FLSD-53. For the exact AUROC scores, please refer to Appendix J. These results further highlight the benefit of an inherently calibrated model produced using AdaFocal as post-hoc techniques such as temperature scaling, as shown in the figure, is ineffective under distributional shift Snoek et al. (2019).

heading: 7 CONCLUSION
text: In this work, we first revisit the calibration properties of regular focal loss and highlight the downside of using a fixed γ for all samples. Particularly, by studying the calibration behaviour of different samples in different probability region, we find that there’s no single γ that achieves the best calibration over the entire region. We use this observation to motivate the selection of γ independently for each sample (or group of samples) based on the knowledge of network’s under/over-confidence. We propose a calibration-aware adaptive focal loss called AdaFocal that accounts for such information and updates the γt at every step based on γt−1 from the previous step and the magnitude of network’s under/over-confidence. We find AdaFocal to perform consistently better across different datasetmodel pairs producing innately calibrated models that most times do not substantially benefit from post-hoc processing of temperature scaling. Additionally, we find models trained with AdaFocal to be significantly better in out-of-distribution detection task.
Reproducibility For reproducibility, we have include in the supplementary material a zip file that contains the code base for running the experiments. For running particular experiments
• CIFAR-10, ResNet-50, Cross entropy: python train.py –dataset cifar10 –model resnet50 –loss cross_entropy –num_bins 15 -e 400 –save-path experiments/cifar10_resnet50_ce
• CIFAR-100, ResNet-50, Cross entropy: python train.py –dataset cifar100 –model resnet50 –loss cross_entropy –num_bins 15 -e 400 –save-path experiments/cifar100_resnet50_ce
• Tiny-ImageNet, ResNet-50, Cross entropy: python train.py –dataset tiny_imagenet –model resnet50_ti –loss cross_entropy –num_bins 15 –first-milestone 40 –secondmilestone 60 -e 100 -b 64 -tb 64 –dataset-root data/tiny-imagenet-200 –save-path experiments/tinyImageNet_resnet50_ce
• 20 Newgroups, CNN, Cross entropy: python main.py –loss cross_entropy –num-epochs 50 –num-bins 15 –save-path experiments/cnn_ce

heading: APPENDICES

heading: A ADAFOCAL’S GENERALIZATION TO LARGE SCALE DATASET (IMAGENET)
text: For ImageNet, FLSD-53 seems to perform very poorly in terms of calibration. The reason is that due to higher values of γ = 5, 3 FLSD-53 becomes extremely under-confident in each bin leading to a high calibration error. AdaFocal, on the hand, remains well calibrated which confirms that during training, unlike CE or FLSD-53, AdaFocal being aware of the network’s current under/overconfidence (through the validation set) is able to adjusts the γs in a way that maintains a well calibrated model at every step. Further, in Figure 8, note the dynamics/evolution of γt in different bins. For ImageNet, we find AdaFocal’s γ → 0 which makes sense because, from Figure 7, cross entropy (i.e. γ = 0 for every bin) is much better calibrated than FLSD-53 and AdaFocal (starting from γ = 1) settles down as CE (γ = 0). Note that for an unseen dataset-model pair it is not possible to know beforehand whether CE or focal loss will perform better. However, from these experiments, we find strong evidence that, for any dataset-model pair, AdaFocal will lead to the γs that result in the best calibration.

heading: B CALIBRATION BEHAVIOUR OF FOCAL LOSS IN DIFFERENT BINS
text: In the main paper, we showed the calibration behavior of different focal losses for ResNet50 trained on CIFAR-10 for only a few bins. For completeness, the rest of the bins and their calibration error Ei = Cval,i − Aval,i are shown in Figure 9 for focal losses with γ = 0, 3, 4, 5. We observe that there’s no single γ that performs the best across all the bins. Rather, every bin seems to have a particular γ that achieves the best calibration.

heading: C CORRESPONDENCE BETWEEN CONFIDENCE OF TRAINING AND VALIDATION SAMPLES

heading: D CALFOCAL LOSS

heading: E DATASETS AND EXPERIMENTS

heading: E.1 DATASET DESCRIPTION
text: CIFAR-10 Krizhevsky (2009): This dataset contains 60, 000 coloured images of size 32 × 32, which are equally divided into 10 classes. A split of 45, 000/5, 000/10, 000 images is used as train/validation/test sets respectively.
CIFAR-100 Krizhevsky (2009): This dataset contains 60, 000 coloured images of size 32 × 32, which are equally divided into 100 classes. A split of 45, 000/5, 000/10, 000 images is used as train/validation/test sets respectively.
ImageNet Russakovsky et al. (2015): ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012-2017 is an image classification and localization dataset. This dataset spans 1000 object classes and contains 1,281,167 training images and 50,000 validation images.
Tiny-ImageNet Deng et al. (2009): It is a subset of the ImageNet dataset with 64× 64 dimensional images and 200 classes. It has 500 images per class in the training set and 50 images per class in the validation set.
20 Newsgroups Lang (1995): This dataset contains 20, 000 news articles, categorised evenly into 20 different newsgroups. Some of the newsgroups are very closely related to each other (e.g. comp.sys.ibm.pc.hardware / comp.sys.mac.hardware), while others are highly unrelated (e.g misc.forsale / soc.religion.christian). We use a train/validation/test split of 15, 098/900/3, 999 documents.

heading: E.2 EXPERIMENTAL DETAILS
text: For all our experiments, we have used Nvidia Titan X Pascal GPU with 12GB memory.
CIFAR-10 and CIFAR-100: We use SGD with a momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the first 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs. We use a training batch size of 128. The training data is augmented by applying random crops and random horizontal flips.
Tiny-ImageNet: We use SGD with a momentum of 0.9 as our optimiser, and train the models for 100 epochs with a learning rate of 0.1 for the first 40 epochs, 0.01 for the next 20 epochs and 0.001 for the last 40 epochs. We use a training batch size of 64. Note that we use 50 samples per class (i.e. a total of 10000 samples) from the training set as the validation set. Hence, the training is only on 90000 images. We use the Tiny-ImageNet validation set as our test set.
ImageNet: We use SGD as our optimiser with momentum of 0.9 and weight decay 10−4, and train the models for 90 epochs with a learning rate of 0.01 for the first 30 epochs, 0.001 for the next 30 epochs and 0.0001 for the last 30 epochs. We use a training batch size of 128. We divide the 50,000 validation images into validation and test set of 25,000 images each.
20 Newsgroups: We train the Global Pooling CNN Network Lin et al. (2014) using the Adam optimiser, with learning rate 0.001, and default betas 0.9 and 0.999. We used Glove word embeddings
Pennington et al. (2014) to train the network. We train the model for 50 epochs and use the model at the end to evaluate the performance.
All our experiments are implemented in PyTorch. The hyperparameters that are not explicitly mentioned above are set to their default values. For CIFAR-10/100 and Tiny-ImageNet, AdaFocal is implemented on top of the base code available from Mukhoti (2020). The code for 20 Newsgroups is implemented in PyTorch by adapting the code (TensorFlow) available from Kumar (2018).
The experimental results in the paper are reported for the model at the end of (1) CIFAR-10/100: 350 epochs, (2) Tiny-ImageNet: 100 epochs, (3) ImageNet: 90 epochs, and (4) 20 NewsGroups: 50 epochs.

heading: F ADAECE AND CLASSWISE-ECE PERFORMANCE
text: Here, we compare the performance of AdaFocal against the baseline methods in terms of AdaECE and classwise-ECE in Table 3 and 4 respectively. For CIFAR-10/100, the values are reported for the model at the end of 350 epochs; for Tiny-ImageNet, at the end of 100 epochs; and for 20 NewsGroup dataset, at the end of 50 epochs. From these tables, we observe that AdaFocal outperforms all the baseline methods by a substantial margin, especially if we compare the pre-temperature scaling results.

heading: Dataset Model Cross Entropy Brier Loss MMCE LS-0.05 FLSD-53 AdaFocal

heading: Dataset Model Cross Entropy Brier Loss MMCE LS-0.05 FLSD-53 AdaFocal

heading: Dataset Model Cross Entropy FLSD-53 AdaFocal

heading: Dataset Model Cross Entropy FLSD-53 AdaFocal

heading: Dataset Model Cross Entropy FLSD-53 AdaFocal

heading: G DEBISED ESTIMATES OF ECE

heading: Dataset Model Cross Entropy FLSD-53 AdaFocal

heading: H ECE ERROR BARS

heading: I NUMBER OF BINS USED DURING ADAFOCAL TRAINING
text: Experiment details: CIFAR-10, ResNet50 trained for 350 epochs. The reported results below are without temperature scaling. Our method AdaFocal with 5, 10, 15, 20, 30, and 50 adaptive (equal mass) bins vs FLSD-53. Note here that there are two types of binning:

heading: J AUROC FOR OUT-OF-DISTRIBUTION DETECTION
text: For ResNet110 on CIFAR-10/SVHN, we were not able to reproduce the reported results of 96.74, 96.92 for FL-3 in Mukhoti et al. (2020). Instead we found those values to be 90.27, 90.39 and report them in Table 10
K MOVING AVERAGE γ-UPDATE RULE
For the focal loss in the paper L(pn, t) = −(1− pn)γb,t log pn, the unconstrained γ-update rule for AdaFocal is given by
γt+1 = γt ∗ exp(Cval,t+1 −Aval,t+1) (5) = γt ∗ exp(Eval,t+1) (6)
If instead we use exponential moving average to update γ, then the update rule (let’s call it MA-α) is given by
γt+1 = (γt) α ∗ ( eEval,t+1 )1−α (7)
= γαt−1 ∗ eαEval,t ∗ e(1−α)Eval,t+1 (8) = γαt−1 ∗ e[αEval,t+(1−α)Eval,t+1] (9)
The evolution or dynamics of γ is given in Figure 16.
L MULTIPLE RUNS OF ADAFOCAL WITH DIFFERENT γmax
Due to the stochastic nature of the experiments, AdaFocal γs may end up following different trajectories across different runs (initialization), which in turn might lead to variations in the final results. In this section, we look at the extent of such variations for (1) unconstrained γ (2) γ capped by γmax = 20 and (3) γ capped by γmax = 50. We study this for ResNet-50 trained multiple times on CIFAR-10 starting with a different random seed.
L.1 ADAFOCAL, γmax = 20
In Figure 17, we observe that AdaFocal with γmax = 20 is consistently (9 out of 9 times) better than FLSD-53. Figure 18 shows the evolution of γ across different runs.
L.2 ADAFOCAL, γmax = 50
In Figure 19, we observe that AdaFocal with γmax = 50 has more variability than AdaFocal γmax = 20 but is mostly better than FLSD-53. Figure 19 shows the evolution of γ across different runs.
L.3 ADAFOCAL, UNCONSTRAINED γ
In Figure 21, we observe that AdaFocal with unconstrained γ does exhibit some variability across different runs: 7 out of 9 times it performs better than FLSD-53 whereas the other two times it is similar or slightly worse.
The above behaviour is mostly due to the variations in the trajectory of γs for lower bins, as shown in Figure 22. For higher bins, we see the γs settling to similar values, however for lower bins, as the γs are unconstrained they blow up to very high values.

heading: M ERROR, ECE AND BIN STATISTICS PLOTS FOR REST OF THE EXPERIMENTS

Review summaries:
This paper studies the calibration of deep learning, which aims to make confidence store accurately describe predictions' correctness probabilities. The authors improve focal loss and propose a calibration-aware focal loss for better calibration.  The proposed approach adaptively adjusts the coefficient of focal loss according to the momentums and current predictions' confidence. The authors conduct experiments on SVNH, CIFAR10/100 datasets to verify the approach's efficacy. 

This paper considers the problem of model calibration. Existing works calibrates the model by post-hoc approaches or objective function tailored for calibration. The authors of the paper propose an adaptive version based on Focal loss, which regularizes the overconfidence of neural networks. They observe that although focal loss improves the calibration, it leaves out the under-confident samples. To mitigate the issue, they propose adjusting the hyper-parameter $\gamma$ in focal loss according to the model's under/over-confidence. Experiments on vision and NLP classification tasks showcase the effectiveness of the adaptive version.