AdaFocal: Calibration-aware Adaptive Focal Loss

1 INTRODUCTION . Neural networks have found tremendous success in almost every field including computer vision , natural language processing , and speech recognition . Over time , these networks have grown complex and larger in size to achieve state-of-the-art performance and they continue to evolve further in that direction . However , it has been well established that such high capacity networks suffer from poor calibration Guo et al . ( 2017 ) , i.e . the confidence scores of the predictions do not reflect the real world probabilities of those predictions being true . For example , if the network assigns 0.8 confidence to a set of predictions , we should expect 80 % of those predictions to be correct . However , this is far from reality since modern networks tend to be grossly over-confident . This is of great concern , particularly for mission-critical applications such as autonomous driving , medical diagnosis , wherein the downstream decision making not only rely on the predictions but also on their confidence . In recent years , there has been a growing interest in developing methods for calibrating neural networks . These can be mainly divided into two categories ( 1 ) post-hoc approaches that perform calibration after training ( 2 ) methods that calibrate the model during training itself . The first includes methods such as Platt scaling Platt ( 1999 ) , histogram binning Zadrozny & Elkan ( 2001 ) , Isotonic regression Zadrozny & Elkan ( 2002 ) , Bayesian binning and averaging Naeini et al . ( 2015 ) ; Naeini & Cooper ( 2016 ) , and Spline fitting Gupta et al . ( 2021 ) . Methods in the second category focus on training the model on an objective function that accounts for calibration as well , including Maximum Mean Calibration Error ( MMCE ) Kumar et al . ( 2018 ) , Label smoothing Müller et al . ( 2019 ) , and recently focal loss Mukhoti et al . ( 2020 ) . These methods aim to produce inherently calibrated models which when combined with post training calibration methods lead to further improvements . Contribution . Our work falls into the second category . We build upon the calibration properties of focal loss to propose a modification that further improves its performance . Firstly , we make the observation that while regular focal loss , with a fixed γ parameter , improves the overall calibration by preventing samples from being over-confident , it also leaves other samples under-confident . To address this drawback , we propose a modification to the focal loss called AdaFocal that adjusts the γ for each training sample ( or rather a group of samples ) separately by taking into account the model ’ s under/over-confidence about a similar corresponding group in the validation set . We evaluate the performance of our method on four image classification tasks : CIFAR-10 , CIFAR-100 , Tiny-ImageNet and ImageNet , and one text classification task : 20 Newsgroup , using various model architectures , and show that AdaFocal substantially outperforms the regular focal loss and other state-of-the-art calibration techniques in the literature . We further study the performance of AdaFocal on an out-of-distribution detection task and find it to perform better than the competing methods . Finally , we find that the models trained using AdaFocal get innately calibrated to a level that most times do not significantly benefit from temperature scaling . 2 PROBLEM SETUP AND DEFINITIONS . Consider a classification setting where we are given a set of training data { ( xn , ytrue , n ) } , with xn ∈ X being the input and ytrue , i ∈ Y = { 1 , 2 , . . . , K } the associated ground-truth label . Using this data we wish to train a classifier fθ ( x ) that outputs a vector p̂ over theK classes . We also assume access to a validation set for hyper-parameter tuning and a test set for evaluating its performance . For example , fθ ( · ) can be a neural network with learnable parameters θ , x is an image , and p̂ is the output of a softmax layer whose kth element p̂k is the probability score for class k. We refer to ŷ = argmaxk∈Y p̂k as the network ’ s prediction and the associated probability score p̂ŷ as the predicted confidence , and the same quantity for the jth example is p̂ŷ , j . In this setting , a network is said to be perfectly calibrated if the predicted confidence p̂ŷ reflects the true probability of the network classifying x correctly i.e . P ( ŷ = ytrue | p̂ŷ = p ) = p , ∀p ∈ [ 0 , 1 ] Guo et al . ( 2017 ) . Continuing our example , if the network assigns an average confidence score of 0.8 to a set of predictions then we should expect 80 % of those to be correct . We define Calibration Error as E = p̂ŷ − P ( ŷ = ytrue | p̂ŷ ) and the Expected Calibration Error as Ep̂ŷ [ E ] = Ep̂ŷ [ |p̂ŷ − P ( ŷ = ytrue | p̂ŷ ) | ] Guo et al . ( 2017 ) . However , as the true calibration error can not be computed empirically with a finite sized dataset , the following three approximations are generally used in the literature . That is , for a dataset { ( xn , ytrue , n ) } Nn=1 , ( 1 ) ECE = ∑M i=1 |Bi| N |Ci − Ai| Guo et al . ( 2017 ) , where Bi is equal-width bin that contains all examples j with p̂ŷ , j in the range [ iM , i+1 M ) , Ci = 1 |Bi| ∑ j∈Bi p̂ŷ , j is the average confidence and Ai = 1 |Bi| ∑ j∈Bi 1 ( ŷj = ytrue , j ) is the bin accuracy . Note that Ei = Ci − Ai is the empirical approximation of the calibration error E , ( 2 ) AdaECE = ∑M i=1 |Bi| N |Ci − Ai| Nguyen & O ’ Connor ( 2015 ) , where ∀i , j |Bi| = |Bj | are adaptively sized ( equal-mass ) bins that contain an equal number of samples , and ( 3 ) ClasswiseECE Kumar et al . ( 2018 ) ; Kull et al . ( 2019 ) estimates the calibration over all K classes : ClasswiseECE = 1 K ∑M i=1 ∑K k=1 |Bi , k| N |Ci , k −Ai , k| where Ci , k = 1 |Bi , k| ∑ j∈Bi , k p̂k , j is the average confidence for the kth class and Ai , k = 1|Bi , k| ∑ j∈Bi , k 1 ( ytrue , j = k ) is the accuracy of the kth class in the ith bin . Lastly , as ECE has been shown to be a biased estimate of true calibration Vaicenavicius et al . ( 2019 ) , we additionally use two de-biased estimates of ECE namely ECEdebiased proposed in Kumar et al . ( 2019 ) and ECEsweep proposed in Roelofs et al . ( 2021 ) to further confirm our results . 3 CALIBRATION PROPERTIES OF FOCAL LOSS . Focal loss Lin et al . ( 2017 ) LFL ( p ) = − ( 1 − p ) γ log p was originally proposed to improve the accuracy of classifiers by focusing on hard examples and down-weighting well classified examples . Recently it was further shown that focal loss may also result in significantly better calibrated models than cross entropy Mukhoti et al . ( 2020 ) . This is because , based on the relation : LFL ≥ KL ( q||p̂ ) − γH ( p̂ ) where q is the one-hot target vector , focal loss while minimising the main KL divergence objective also increases the entropy of the prediction p̂ . As a consequence this prevents the network from being overly confident on wrong predictions and overall improves calibration . The regular focal loss with fixed γ , as we show in this section , does not achieve the best calibration . In Figure 1 , we plot the calibration behaviour of ResNet50 in different bins when trained on CIFAR-10 with different focal losses . The ith bin ’ s calibration error subscripted by `` val '' Eval , i = Cval , i−Aval , i is computed on the validation set using 15 equal-mass binning . The figure shows the lowest ( bin-0 ) , a middle ( bin-7 ) and highest bin ( bin-7 ) . For reference , the rest of the bins and their bin boundaries are shown in Appendix B . From Figure 1 ( a ) , we see that although focal loss γ = 4 achieves the overall lowest calibration error ( AdaECE ) , there ’ s no single γ that performs the best across all the bins . For example , in bin-0 γ = 4 , 5 seems to achieve better calibration whereas γ = 0 , 3 are over-confident . For bin-7 , on the other hand , γ = 3 seems to be better calibrated whereas γ = 4 , 5 are under-confident and γ = 0 is over-confident . This clearly indicates that using different γs for different bins can further improve the calibration . Such an attempt is presented in Mukhoti et al . ( 2020 ) called the Sample-Dependent Focal Loss ( FLSD-53 ) which assigns γ = 5 if the training sample ’ s true class posterior p̂ytrue ∈ [ 0 , 0.2 ) and γ = 3 if p̂ytrue ∈ [ 0.2 , 1 ] . However , this strategy is fixed for every dataset-model pair and is based on simple heuristics of choosing higher γ for smaller values of p̂ytrue and relatively lower γ for higher values of p̂ytrue . However , from Figure 1 ( b ) , we see that FLSD-53 is also not the best strategy across all the bins . This , therefore , motivates the design of a γ selection strategy that can assign an appropriate γ for each bin based on the magnitude and sign of Eval , i . However , in order to design such a strategy we need solutions to the following two major challenges : 1 . How do we find some correspondence between the `` confidence of training samples '' , which we can manipulate during training by adjusting the entropy regularising parameter γ , and the `` confidence of the validation samples '' , which we want to be actually manipulated but do not have direct control over ? In other words , in order to indirectly control the confidence of a particular group of validation samples , how do we know which particular group of training samples ’ confidence to be manipulated ? 2 . Given that there is a correspondence between a training group and a validation group ( even if it ’ s loose ) , how do we arrive at the exact values of γ that will lead to better calibration ? We try to answer the first question in the next section and the answer to the second question leads to AdaFocal which is the main contribution of the paper . 4 CORRESPONDENCE BETWEEN CONFIDENCE OF TRAIN AND VAL . SAMPLES . In order to find some correspondence , an intuitive thing to do would be to group the validation samples into M equal-mass validation-bins , and then use these validation-bin boundaries to group the training samples as well . Then , we can compare the average confidence of the validation samples and the average confidence of the training samples , in the same validation-bin , to check for any correspondence . Quantities of interest For binning validation samples , we always look at the confidence of the top predicted class ŷ denoted by p̂val , top ( bin average : Cval , top ) . For training samples , on the other hand , instead of the confidence of the top predicted class ŷ denoted by p̂train , top ( bin average : Ctrain , top ) , we will focus on the confidence of the true class ytrue denoted by p̂train , true ( average : Ctrain , true ) because during training we only care about p̂train , true which is manipulated through some loss function . For reference however , Figure 10 in Appendix C compares Ctrain , true and Ctrain , top to show that as the training set accuracy approaches 100 % , the top predicted class and the true class for a training sample become the same . Henceforth , for a cleaner notation , we will always refer to Ctrain ≡ Ctrain , true and Cval ≡ Cval , top . Common binning When training samples are grouped using the bin boundaries of the validationbins . In Figure 2 ( b ) , we compareCtrain , i in validation-bin-i 1 withCval , i in the same validation-bin-i , and find that there is indeed a good correspondence between the two quantities . For example in Figure 2 ( b ) , as γ increases from 0 , 3 to 5 , the solid-line ( Ctrain , i ) gets lower , and the same behaviour is observed on the starred-line ( Cval , i ) as well . For completeness , rest of the bins are shown in Figure 12 Appendix C. This is very encouraging as now we can expect ( even though loosely ) that if we increase/decrease the confidence of a group of training samples in some lower ( or middle , or higher ) probability region then the same will be reflected on a similar group of validation samples in lower ( or middle , or higher ) probability region . This therefore provides a way to indirectly control the value of Cval , i by manipulating Ctrain , i , and from a calibration point of view , our strategy going forward would be to exploit this correspondence to keep Ctrain , i ( which we have control over during training ) closer to Aval , i ( the validation set accuracy in validation-bin-i ) so that Cval , i also stays closer to Aval , i to overall reduce the calibration error Eval , i = Cval , i −Aval , i . Independent binning Before proceeding , for completeness , we also look at the case when training samples and validation samples are grouped independently into their respective training-bins and validation-bins . Figure 2 ( a ) compares Ctrain , i in training-bin-i with Cval , i in validation-bin-i . We observe a similar behaviour as mentioned above . Note that since the binning is independent , the boundaries of training-bin-imay not be exactly the same as that of validation-bin-i , however as shown in Figure 11 Appendix C ( along with rest of the bins and their bin boundaries ) , they are quite close , meaning that a training group in lower ( /middle/higher ) probability region have good correspondence with the validation group in a similar nearby region . Going forward , for the ease of algorithm design , we will simply stick to the case of `` common binning '' where training samples are grouped as per validation-bin boundaries . This will allows us to maintain a one-to-one correspondence between the boundaries of the ith training and validation group .

Summary: This paper considers the problem of model calibration. Existing works calibrates the model by post-hoc approaches or objective function tailored for calibration. The authors of the paper propose an adaptive version based on Focal loss, which regularizes the overconfidence of neural networks. They observe that although focal loss improves the calibration, it leaves out the under-confident samples. To mitigate the issue, they propose adjusting the hyper-parameter $\gamma$ in focal loss according to the model's under/over-confidence. Experiments on vision and NLP classification tasks showcase the effectiveness of the adaptive version.