AdaFocal: Calibration-aware Adaptive Focal Loss
Authors: Arindam Ghosh
Thomas Schaaf
Matthew R. Gormley
Venue: ICLR 2022 Submitted (2022)
Keywords: neural networks
uncertainity calibration
out of distribution detection

Abstract: Much recent work has been devoted to the problem of ensuring that a neural network's confidence scores match the true probability of being correct, i.e. the calibration problem. Of note, it was found that training with Focal loss leads to better calibrated deep networks than cross-entropy loss, while achieving the same level of accuracy \cite{mukhoti2020}. This success stems from Focal loss regularizing the entropy of the network's prediction (controlled by the hyper-parameter $\gamma$), thereby reining in the network's overconfidence. Further improvements in calibration can be achieved if $\gamma$ is selected independently for each training sample. However, the proposed strategy (named FLSD-53) is based on simple heuristics which, when selecting the $\gamma$, does not take into account any knowledge of whether the network is under or over confident about such samples and by how much. As a result, in most cases, this strategy performs only slightly better. In this paper, we propose a calibration-aware sample-dependent Focal loss called AdaFocal that adaptively modifies $\gamma$ from one training step to the next based on the information about the network's current calibration behaviour. At each training step $t$, AdaFocal adjusts the $\gamma_t$ based on (1) $\gamma_{t-1}$ of the previous training step (2) the magnitude of the network's under/over-confidence. We evaluate our proposed method on various image recognition and NLP tasks, covering a variety of network architectures, and confirm that AdaFocal consistently achieves significantly better calibration than the competing state-of-the-art methods without loss of accuracy.

Under review as a conference paper at ICLR 2022
ADAFOCAL:
CALIBRATION-AWARE ADAPTIVE FOCAL Loss
Anonymous authors
Paper under double-blind review
ABSTRACT
Much recent work has been devoted the problem of ensuring that neural network's confidence scores match the true probability of being correct, i.e. the calibration problem. Of note, it was found that training with focal loss leads better calibrated deep networks than cross-entrops while achieving the same level of accuracy Mukhoti al, (2020). This success stems from focal loss regularizing the entropy of the network's 's prediction (controlled by the thereby reining in the network's 's overconfidence. Further improvement is expected if y is selected independently for each training sample. However, the proposed Sample-Dependert Focal Loss (FLSD) in Mukhoti et al. (2.20) is based on simple heuristics that does not take into account the difference in the network's k's calibration behaviour for different samples (or groups of samples). As a result it is only slightly better than focal loss with fixed 7. In this paper, we propose calibration-aware version of FLSDD, called AdaFocal, which, at every training step i, adaptively modifies the y for individual group of samples based on (1) 7-1 from the previous training step (2) the magnitude of the network's ' under/over-comfidenne for those groups. We evaluate our method on various small to large-scale image recognition tasks and one NLP task, covering a variety of network architectures, to confirm that AdaFocal consistently achieves improved calibration without a significant loss in accuracy. Further, the models trained with AdaFocal are shown to have significantly improved cattof-DDitribution (OOD) detection capability.
7),
INTRODUCTION
Neural networks have found tremendous success in almost every field including computer vision, natural language processing, and speech recognition. Over time, these networks have grown complex and larger in size to achieve state-of-the-art performance and they continue to evolve further that direction. However, it has been well established that such high capacity networks suffer from poor calibration Guo et al. (2017). i.e. the confidence scores of the predictions do not reflect the real world probabilities of those predictions being true. For example, if the network assigns 0.8 confidence to a set of predictions, we should expect 80% of those predictions to be correct. However, this is far from reality since modern networks tend to be grossly over-confident. This is of great concern, particularly for cession-critica applications such as autonomous driving, medical diagnosis, wherein the downstream decision making not only rely on the predictions but also on their confidence.
In recent years, there has been a growing interest in developing methods for calibrating neural networks. These can be mainly divided into two categories (1) post-hoc approaches that perform calibration after training (2) methods that calibrate the model during training itself. The first includes methods such as Platt scaling Plati (199)), histogram binning Zadrozny & Elkan (200T), Isotonic regression Zadrozny & Elkan 2002). Bayesian binning and averaging Naeini al. (2013); )); Naeini & Cooper 2016), and Spline fitting Gupta et al. (2021). Methods in the second category focus on training the model on an objective function that accounts for calibration well, including Maximum
Mean Calibration Error (MMCE) Kumar et al. (2018), Label smoothing Müller er al.) (2019), and recently focal loss Mukhoti et al. (202)). These methods aim produce inherently calibrated models which when combined with post training calibration methods lead to further improvements.
Contribution. Our work falls into the second category. We build upon the calibration properties of focal loss propose modification that further improves its performance. Firstly, we make the observation that while regular focal loss, with a fixed y parameter, improves the overall calibration by preventing samples from being over-confident, it also leaves other samples under-confidentt. To

---

Under review as a conference paper at ICLR 2022
address this drawback, we propose a modification to the focal loss called AdaFocal that adjusts the y for each training sample (or rather a group of samples) separately by taking into account the model's underr/oeer-confidenne about a similar corresponding group in the validation set. We evaluate the performance of our method on four image classification tasks: CIFAR-10, 10, CIFAR-100, Imn and ImageNet,, and one text classification task: 20 Newsgroup, using various model architectures, and show that AdaFocal substantially outperforms the regular focal loss and other state-of-the-an calibration techniques in the literature. We further study the performance of AdaFocal on an out-of- diftrriiiiuttonn detection task and find it to perform better than the competing methods. Finally, we find that the models trained using AdaFocal get innately calibrated to a level that most times do not significantly benefit from temperature scaling.
2 PROBLEM SETUP AND DEFINITIONS
Consider a classification setting where we are given set of training data {(x,, (true,n) }} with Xn E X' being the input and Yirue,i E y = 21, K} the associated ground-truth label. Using this data we wish to train a classifier fx) that outputs a vector P over the K classes. We also assume access to a validation set for hyper-parameter tuning and a test set for evaluating its performance. For example, (a(-) can be neural network with learnable parameters 0, x is an image, and p is the output of a softmax layer whose hthh element 2 is the probability score for class k. We refer to ÿ = arg maxkey Pk the network's prediction and the associated probability score Pg as the predicted confidence, and the same quantity for the jth example is Pg.j-
In this setting, network is said be perfectly calibrated if the predicted confidence P3 reflects the true probability of the network classifying x correctly i.e. P(ù = ytrue P3 = p) = p. vp E [0, Guo et al. (0177) Continuing our example, if the network assigns an average confidence score of 0.8 to set of predictions then we should expect 80% of those to be correct. We define Calibration Error as & Pg P(ğ J/true i'g) and the Expected Calibration Error as Eg, [
Egg 18g P Utrue pg)] Guo al. (017). However, as the true calibration error cannot be computed empirically with a finite sized dataset, the following three approximations are generally
N IC; All Guo et al. (0117), where B, is equal-width bin that contains all examples j with Pa.s in the range A M C.C. B. Ejen, ; Pā.j is the average confidence and A, = B. Ejen, n. [ = 3hrue..) is
used in the literature. That is, for a dataset ((xn;Brrue.n) Utrue.n)
J (1) ECE > [B
the bin accuracy. Note that Eg = C;- - A, is the empirical approximation of the calibration error
(2) AdaECE- > M Bdl IC;- A.l Nguyen & O'Connor (2015), where Vi, [B:] |B are i-1 adaptively sized (equal- mass) bins that contain an equal number of samples, and (3) ClasswiseECE Kumar et (2018); Kull et al. (2019) estimates the calibration over all K classes: ClasswiseECE
[6.. IC Ar, where C. [ ΣjEB,, Pk.j is the average confidence for the ktth class and AI, B., ΣjEB,, [ = k) is the accuracy of the kth class in the ith bin.
*
EE
M
Ek=1
Lastly, as ECE has been shown to be a biased estimate of true calibration Vaicenavicius et al. (2019). we additionally use two de-biased estimates of ECE namely ECEdebiased proposed in Kumar et al. (2019) and ECEgwoep proposed in Roelots et al. (202I) to further confirm our results.
3 CALIBRATION PROPERTIES OF FOCAL LOSS
Focal loss Lin et al. 2017) (F)(p) = - 7 logp p was originally proposed to improve the accuracy of classifiers by focusing on hard examples and dewn-weighting well classified examples. Recently was further shown that focal loss may also result in significantly better calibrated models than cross entropy Mukhoti et al. (2020). This is because, based on the relation: LF, > KT.(gllp) - (p) where g is the one target vector, focal loss while minimising the main KI. divergence objective also increases the entropy of the prediction p. As consequence this prevents the network from being overly confident on wrong predictions and overall improves calibration.
The regular focal loss with fixed 7, as we show in this section, does not achieve the best calibration. In Figure I we plot the calibration behaviour of ResNet50 in different bins when trained on CIFAR-10 with different focal losses. The ith bin's 's calibration error subscripted by "val" Eval, Cval, - A is computed on the validation set using 15 equal-mass mass binning. The figure shows the lowest (bin-O), a middle (bin-7) and highest bin (bin-7). For reference, the rest of the bins and their bin boundaries are
2

---

Under review as a conference paper at ICLR 2022
Bin.0
0.02
Bin-7
Bin-14
Cvali Aval.
0.
FL-3
1 MINECE 190)
FL-4
0,000
FL-5
0.3
0.07
FL-53
0.000
0.04
100
(a) &eAdBECE vs. epoch
Tpoch
(b) Eval, -ouul Avat, vs. epoch
Epoch
Spoch
Figure 1: Calibration behaviour of ResNet-50 trained on CIFAR-10 with cross entropy (CE), focal
loss y = 3, 4, 5 (FL-3/4/5) and FLSD-53 3 (or FL-53). The statistics are computed using 15 equal-mass binning on the validation set. (a) AdaECE, and (b) Calibration error Eval, Crai. Aval, for lower (bin-0), middle (bin-7), and upper (bin- 14) bin. The black horizontal lines in (b) represent
Feat,; = These exemplify the downside of regular focal loss i.e. although FL.4 achieves the overall lowest calibration error (AdaECE), the best performing 7 is different for different bins.
shown in Appendix B From Figure (a), we see that although focal loss = 4 achieves the overall lowest calibration error (AdaFCE), there's 's no single y that performs the best across all the bins. For example, in bin-0 = 5 seems to achieve better calibration whereas 7 = 0,3 3 are over-confident.
For bin-7, on the other hand, y = 3 seems to be better calibrated whereas 7 = 4, are under-confident and 7 = 0 is over-confident.
This clearly indicates that using different ys for different bins can further improve the calibration. Such attempt is presented in Mukhoti et al. (2020) called the Sample-Depender Focal Loss (FLSD-53) which assigns 7 = 5 the training sample's 's true class posterior Pyrrue E [0. 0.2) and y y= if Porres E [0.2, 1]. However, this strategy is fixed for every dataset-model pair and is based on simple heuristics of choosing higher y for smaller values of pyrrus and relatively lower y for higher values of P However, from Figure [b))
see that FLSD-53 is also not the best strategy across all the bins. This, therefore, motivates the design of a selection strategy that can assign an appropriate y for each bin based on the magnitude and sign of Emat, However, in order to design such a strategy we need solutions to the following two major challenges:
1. How do we find some correspondence between the "confidence of training samples", which we can manipulate during training by adjusting the entropy regularising parameter y, and the "confidence of the validation samples", which we want to be actually manipulated but do not have direct control over? In other words, in order indirectly control the confidence of a particular group of validation samples, how do we know which particular group of training samples' confidence be manipulated?
2. Given that there is a correspondence between training group and a validation group (even if it's loose), how do we arrive at the exact values of fy that will lead to better calibration?
We try to answer the first question in the next section and the answer the second question leads to AdaFocal which is the main contribution of the paper.
4 CORRESPONDENCE BETWEEN CONFIDENCE OF TRAIN AND VAL.. SAMPLES
In order to find some correspondence, an intuitive thing to do would be to group the validation samples into M equal-mass validation-bins and then use these validation-bin boundaries to group the training samples as well. Then, compare the average confidence of the validation samples and the average confidence of the training samples, in the same validation-bin, to check for any correspondence.
Quantities of interest For binning validation samples, we always look at the confidence of the top predicted class ý denoted by Peal (bin average: Coal.top). For training samples, on the other hand, instead of the confidence of the top predicted class ÿ denoted by Ptrain,top (bin average: Ctrain (to)) we will focus on the confidence of the true class 4/rue denoted by Perain,true (average: Ctrain,true) because during training we only care about Ptrain.true which is manipulated through some loss function. For reference however, Figure 10 in Appendix compares Ctrain,true and Cerain,ton to show that the training set accuracy approaches 100%, the top predicted class and the true class for a training sample become the same. Henceforth, for a cleaner notation, we will always refer to Ctrain Crrain.trus and Cral Crat,top-
3

---

Under review as a conference paper at ICLR 2022
Training/Validation bin-0
Training bin-7
Training/Vlliatiion bin-14
1.0
1.000
Avg. Confidence
Cewn
CE Gua
TU3, Cree
0.975
0.950
FLO, Ca
0.70
100
200
300
100
200
300
0.900
100
200
300
Epech
Epoch
Epoch
(a) Independent binning: training samples and validation samples are grouped independently into training-bins and validation-bins respectively, Solid line: Cerain. and Starred line: Cra!-
Validation bin-D
Validation bin-7
Validation bin-14
1.0
1.000
Avg. Confidence
Cawe
CE Cre
0.8
TUO, Cran
0.975
nLa,
0.950
Cur
0.925
0.7
0
100
200
300
100
200
300
0.900
01
100
200
300
Epoch
Epoch
Epoch
(b) Common binning: training samples grouped using validation-bie bin boundaries. Solid: Ctrain. Starred Coa!
Figure 2: Correspondence between average confidence of a group of training samples (Ctrain) and a group of validation samples (Ceat) for ResNet-50 trained on CIFAR- 10 with focal loss 7 = 0 (CE), 3, The binning involves 15 equal-mass bins with a lower (bin-O), middle (bin-7) and upper (bin-14) shown here. We see good correspondence between Crrain (solid lines) and C.at (starred lines).
Common binning When training samples are grouped using the bin boundaries of the validation- bins. In Figure we compare Ctrain,i in validation -anatiooonnn--leeee with Ceat,s in the same validation bin-i, and find that there is indeed good correspondence between the two quantities. For example in Figure b)), as y increases from 0, 3 to 5, the solid-line (Ctrain, gets lower, and the same behaviour observed on the starred-line (Cool.s) as well. For completeness, rest of the bins shown in Figure 12 Appendix C This is very encouraging as now we can expect (even though loosely) that if we increase/decrease the confidence of a group of training samples in some lower (or middle, or higher) probability region then the same will be reflected on a similar group of validation samples in lower (or middle, or higher) probability region. This therefore provides a way to indirectly control the value of Ceat.; by manipulating Ctrain,,, and from a calibration point of view, our strategy going forward would be to exploit this correspondence to keep Cirain, (which we have control over during training) closer to A (the validation set accuracy in validation-bin-i) so that Cwat,i also stays closer to Aval, to overall reduce the calibration error Eval, = Cval, Acal,i.
Independent binning Before proceeding, for completeness, we also look at the case when training samples and validation samples are grouped independently into their respective training-bins and validation-bins. Figure[[a) compares Ctrain, in training-bin-i with Cood. in validation- observe a similar behaviour as mentioned above. Note that since the binning is independent, the boundaries of training-bin may not be exactly the same as that of palidation-bin-1 however as shown in Figure I1 Appendix C (along with rest of the bins and their bin boundaries), they are quite close, meaning that a training group in lower (/middle/higher) probability region have good correspondence with the validation group in similar nearby region.
We
Going forward, for the case of algorithm design, we will simply stick to the case of "common binning" where training samples grouped as per validation-bin bin boundaries. This will allows us to maintain a one-to-one correspondence between the boundaries of the ith training and validation group.
5 PROPOSED METHOD
Let's denote the nthh training sample's true class posterior Pounce by Pa- Given that Pn. falls into validation-bin-b, our goal is to keep Pn, or as per the discussion above its averaged equivalent Cirain, b. closer to A so that the same is reflected on Cyal, b. For manipulating Pa. we will utilize the regularization effect that focal loss's parameter has on the confidence of the predictions Mukhoti
It may happen that no training sample belong to particular validation-bin's boundaries. In that case. Ctrain.s has been shown to drop to zero for example in bin-14 in Figure (b).
4

---

Under review as a conference paper at ICLR 2022
et al. (2020). At this point, can choose to update To either based (1) how far Pn is from Aval,b ie. 7 = f(pn- Avat,b) or (2) how far Ceal,b is from Avat, i.c. y = f(Cool.b - A Such y-update-rule should ensure that whenever the model is over-confident, i.c. Pn. A 5 (or Geat.b > A Ava,b)) , y is increased so that the gradients get smaller which prevents Pa, from increasing further. On the other hand, when Pr. b (or Cral,b < Aval,b), i.e. the model is under-confident, decrease y so to get larger gradients that in turn will increase pu.
Based on this discussion, next we design and study a calibration calirratoon-wwre aware y-update strategy called CalFocal, which with some additional modifications lead to AdaFocal.
5.1
CALIBRATION AWARE FOCAL LOSS (CALFOCAL)
0.5
('ecsueery
1.0
CE
20
0.8
CT-10
Ch-120
0.3
A 100
15
ECE 1%5
0.6
CE
TL5D-53
FLSD-53
CF-10
CT-100
08
100
200
300
200
Epach
Epcch
(a) CalFocal, Avat,b 0.8
Error
(c) ECE (%)
(d) Cerain (solid), Ceat (starred), Aval (dashed)
Figure 3: ResNet-50 trained on CIFAR-10 with cross entropy (CE) and Focal (CF-X). Sub-figure (d) compares Ctrains Crat and Avol in validation bin-O to show that as CalFocall tries to keep Ctroin closer to Aval, Cval also gets closer to Agai-
Case 1: 7 f(m - Avat,b) Treating Avat,b as the point that we want Pr. to not deviate from, we make the focal loss parameter y a function pr. Avai,b to get
Fo (pr) -(1 Pa) logPn:
with, Yn- exp(A(pp Aval,))),
(1)
where, is the validation-bin in which pn falls. The hyper-parameter À is the scaling factor which combined with the exponential function helps to quickly ramp up/down y. The exponential function adheres to the 7-update rule mentioned earlier and also ensures 7 is >> 0. Figure 3|a) plots Foat vs. Pr, for Aval, 0.8. We see that based on the strength of X, the loss drastically drops near Pa = 0.8 and thereafter remains close to zero. This shows that LCalFocul aims is to first push towards 0.8 and then slow its growth towards overconfidence. Next, Figure 3c), we find that CalFocal with À = 10, 100 is able to reduce the calibration error compared to cross entropy but it is still far from FLSDD-53's performance. Also note in Figure 3b) that too high (=100) affects the accuracy of model. Most importantly, Figure3]d) compares Ctroin,i with Cuat, (and also Aoat,) .) for bin-O, where we find some evidence that the strategy of bringing Pn or Cerain, (solid lines) closer to A (dashed lines) results in Coal, (starred lines) getting closer Aval, well, thus reducing the calibration error Eval, = Cval,i - Aval,s slightly.
Case 2: y = f(Cral,b Aval,)) Note that Eq. asssggns a different 7/ for each training sample. To reduce computation and avoid using a different 7n for each training sample, one can instead use a common nt, for all the training samples that fall into the validation-bin b by simply making it a function of Ceat. Avat.b instead of Pn Avat.b-
CcalFocal = -(1 -1-20 (1-pa)" )m log Pn.
To
with, 76 = exp(A(Crel. - Aval,)))
(2)
where, b is the validation-bin in which Pn falls. As shown in Appendix D it's performance is very similar (or slightly better than) CalFocal in Eq. [ Further, it makes more sense to update y based on how far Cvel,b is from A instead of how far Pa. is from A because, as shown in Figure 3 d) bin-O, one may find Cval.b (starred lines) quite closer to A (dashed lines) even when Pa, or its
NNot that for focal loss increasing y does not always lead smaller gradients. This mostly holds true in the region p., approximately > 0.2 (see Figure 3(a) in Mukhoti et al. (2020)) However, in practice and as shown by the training- bin boundaries of bin-O and bin- in Figure Figure Appendix[[] we find majority of the training samples to lie above 0.2 during the majority of the training. and therefore, for the experiments in this paper, we simply stick to the rule of increasing to decrease gradients and stop Pr. from increasing and vice versa.
5

---

Under review as a conference paper at ICLR 2022
average equivalent Ctrain (solid lines) is far from Avat,b At this point when Ceat,b Aval,b. should stop updating further, even though pn - -Aaat à + 0, as we have reached our goal of making use Eq. of Case as base for AdaFocal.
Exat, &rat,b A = O. Therefore,
Limitations of CalFocal: (1) Let's say at some point of training, a high Tb over the next few epochs reduces the calibration error Eval,b = Call,b Avai,b. Then, it is desirable to continue the training with the same high 7h. However, note CalFocal's update rule in Eq. 2 which will reduce - 1 as the G - Avel.b + 0. (2) At some point let's say Cval.E - A is quite high. This will set 76 to some high value as well depending on the hyper-parameter A. Assuming this T6 is still not high enough to bring down the confidence, we would want a way to further increase "o- However, CalFocal is incapable of doing so as it will continue to hold at 7 exp(X(vval, Aeal.b)). By addressing these two issues in the next sub-section we present the final algorithm for AdaFocal.
5.2 CALIBRATION- AWARE ADAPTIVE FOCAL LOSS (ADAFOCAL)
straightforward way to address the above limitations is to make ya. depend on 7b,1-1 i.e.
[(pn: t) = (1 (1- 11-pph 7h,) log Pn.
with 16,4 = 16 exp(Ceal,b - Aeal,)).
(3)
This update rule address the limitations of CalFocal in the following way. Let's say at some point we observe over-confidence i.e. Evat,b = Gral.b Amat, > 0. Then, in the next step "A will be increased. In the subsequent steps, it will continue to increase unless the calibration error F starts decreasing (this additional increase in y was not possible with CalFocal). At this point, if we find E to start decreasing. that would reduce the increase in t over the next epochs and % will ultimately settle down to a value when Feat, = O (CalFocal at Evel,b = 0 will cause y to go down to 1). Next, if this current value of "/6 starts causing under-confidence i.e. Cval,b Aval,b 0, then the update rule will kick in to reduce y thus allowing CoaL. & to be increased back to Apol, This oscillating behaviour of AdaFocal around the desired point of Cral,b A is its main adavantage in reducing calibration error in every bin. Additionally, also note the absence of the hyper-parameter in the exponent of Eq. B which makes AdaFocal hyper-parameter free.
Finally, note an undesirable property of Eq. 3 which is the unbounded exponential update. This may easily cause ?/ explode as it can be expanded as 7t Tt- exp(Eval,t) Yo exp( + Evat,1 + E + Evaa.:). Thus if Evai,t > 0 for quite a few number of epochs, Y/ will become so large that even if Eval,t < 0 in the subsequent epochs, it may not decrease desired level. We remedy this by simply constraining "t to upper bound Tmax to get the AdaFocal loss as
CAd t) = -(1 -
(1.))) 7o. log with, 7b. = min (mmax: 7b,t- * Cout, - Acalib (4)
An algorithmic description of training with AdaFocal (or CalFocal) is given in Algorithm | Limita- tion: One may argue that Ymax is again a syper-parameter however, note that it does not require any special fine-tuning. Its sole purpose is to stop y from exploding and any reasonable value around 20 works quite well in practice. For all our experiments, we use Tmax = 20. For comparison of AdaFocal with 7max = 20, Ymax 50 and unconstrained Tmmax 0o, please refer to Appendix D
6 EXPERIMENTS
Experimental setup We evaluate the performance of our proposed method on image and text classification tasks. For image classification, we use CIFAR- 10, CIFAR- Krizhevsky (2009).
Deng Image Deng et al. (2009) and ImageNet Russakovsky et al. (015) to analyze the calibration of ResNet50, ResNet- 100 He et al. (2016). Wide-ResNeet-66 10 10 Zagoruyko & Komodakis (2016),
and DenseNet-122 121 Huang et al. (2017) models. For text classification, we use the 20 Newsgroup (ang (1995) and train the Global Pooling CNN model Lin et al. (2014). Further details about the datasets, models and experimental configurations are given in Appendix
Baseline As bascline calibration methods use MMCE Kumar et al. 20118) Brier loss Brier (1950). Label smoothing Mülle et al. (2019) and sample-dependent focal loss FLSD-53, We also report the effect of temperature scaling Cuo et al. (2017) top of these calibration methods. Following Mukhoti et al. (2020), we select the optimal temperature that produces the minimum ECE on the validation set by searching in the interval (0, 10] with step size of 0.1.
Results. In Figure we compare AdaFocal against cross entropy (CE) and FLSD-53 for ResNet-50 trained various small to large-scale image datasets. We chose FL-SD)-53 our competitive baseline

---

Under review as a conference paper at ICLR 2022
Algorithm 1: CalFocal, AdaFocal
Input:
Nerasn and Deal {xx, Yorue, ) Woaa Devasn [(X., Virue, )} n-1
Initialization at = 0: for i to M do [1-1 Beal, M Mr]
|/ equally spaced validation- mid point the bin;
Co Avat,t+1, 2M
75,2 =
Training: for t = 0 to do
L 0:
for to Nerii do ps. Jwn (Xn)
I/ denoting Ps. Pytracm validation-bia inside which p lies: // use "e, 6th to compute loss;
10
b get index(pa. (Ba,)) L. += (1 Pn) 71, logp
11
12
We+1 -raadient_udaateee
L.);
13
for
Using updated model fuo4: on Dual, update bin statisties and y Re-compute bin boundarics B and corresponding 1, A if CalFocal then
M do
14
15
Tt+1,i exp(A (Ceat,et1. Aval,t+is)):
else if AdaFocal then
18
7044-, min (?maxx Yes * coxp(Crat,e41, A
10-
28.
15
ECE 196)
Eror (9)
26
Enror
CF
24
CE
FLSD-53
FLSD-53
AdaFocal
AdaFocal
13c
200
300
200
Epech
Epoch
Epoth
Epoch
(a) CIFAR-10
(b) CIFAR 100
Fl 10-53
FLSD-53 AdaFora
15
ErOr 196)
Adaircal
ECE (0g)
Error (Sc
ECE (%6
20
(0
60 700
25g
Spoch
Spoch
Cnoch
Cooch
(c) InngeeeteNet
(d) ImageNet
Figure 4: Test set error and calibration of ResNet-50 trained on small large-scale image datasets with cross entropy (CE), FLSD-53 and AdaFocal. In each subfigure Left: Error (%), Right: ECE (%). The plots have been averaged over runs. AdaFocal consistently achieves low calibration error across all datasets while maintaining the accuracy.
as it shown be consistently better than MMCE, Brier Loss and Label smoothing Mukhoti iet al. (220 across many datasets-model pairs. The figure plots the test set eтror and ECE calibration metric. In Figure 5 for ResNet-50 on CIFAR-I0 and ImageNet, we plot (1) the calibration statisties Epat = Coal Aval of the validation set and (2) the dynamics of associated used by AdaFocal during the training for few bins covering lower, middle, and higher probability regions.
From these figures, we first observe that for CIFAR- CIFAR- 100 and Tiny Imngeema,,Net, FLDS-53 much better calibrated than CE. This is because, as shown in Figure 5(aa) for ResNet-50 and CIFAR- 10, CE is over-confident compared to FL-SD-53 in every bin. For ImageNet, however, the behaviour is reversed: FL.SD-53 is poorly calibrated than CE. The reason, as shown in Figure(5[bb) that due to the use of high values 7 5, 3, FLSD-53 makes the model largely under-confident in each bin, leading to an overall high calibration error. This shows that FLSD-53 is a strategy based on heuristic (from a limited number of dataset-model pairs) that does not generalize well. AdaFocal, the other hand, is well calibrated for all the four dataset-model pairs while achieving similar accuracy.
The dynamics/evoluttin of 7 during training for different bins is shown in Figure 5 (1) for CIFAR- 10, we find Y to be closer to for higher bins and closer to 20 for lower bin. These ys found by AdaFocal result in better calibration than 7 5,3 of FLSDD-5. (2) for ImageNet, find AdaFocal's
7

---

Under review as a conference paper at ICLR 2022
Bin
Bin
Bin 14
10
0.01
0.01
0.30
0.10
Cyal - Aval
Aval
0.05
0.05
0.00
0.00
Adatocal
0.00
0.00
FLSD-53
-0.01
FLSD-53
AdeForia
Adaiaral
-0.05
-0.05
-0.02
100
-0.30
-0.10
1.4
1.05
AdiFocaa
1.0
AdoFocal
0.95
0.5
1.1
Yt. Aderocal
1.0
Ys. AdaTocal
0.90
Bin
Bin
Bin 6
Bin 0
0.10
0.10
0.2
0.2
Cual - Aval
Avai
0.00
0.00
0.0
0.0
-0.05
-01
...0
200
0.10
100
100
-0.0
40
80
0.20
80
AdaFacal
AdaFocal
1.0
Adaleca
1.0
Yn. Adaffcal
1.2
2.0.
0.0
0.0
0.0
40
Epoch
Epoch
Epoch
Epach
(a) CIFAR-10
(b) ImageNet
Figure 5: Dynamics of y in different bins and calibration statistics of validation set used by AdaFocal
for ResNet-50 trained CIFAR-10 and ImageNet. Each bin has two subplots: Top: Fraa
Coat Aval. Bottom: evolution of %/. Black dotted line in top plot represent zero calibration error. We observe that AdaFocal is able find the ys that produces the lowest calibration in each bin. For CIFAR-10, 7 > 1 whereas for ImageNet ? + because, from Figure 4 we see that cross entropy
(7 0) is better calibrated on ImageNet.
- 0. This makes sense because for ImageNet, from Figure 4 d), cross entropy (i.e. Y for every bin) is much better calibrated than FLSD-53 and AdaFocal (starting from ? = 1) also ultimately settles down to CE (7 0) achieve similar level of calibration. This confirms that during training, unlike CE or FLSD-53, AdaFocal being aware of the network's 's current under/over-confidene to guide the ys to the values that maintain a well calibrated model at every step. Also note that for an unseen dataset-mode pair there's 's no way to know beforehand which will perform better but these empirical evidence show that AdaFocal will automatically find those appropriate ys. Rest of the experiments are shown in Table (ECE) and Table (Error From Table we observe that prior to temperature scaling AdaFocal outperforms the baseline methods by substantial margin in 9 out of 11 cases. With post-temperature scaling included, AdaFocal achieves the lowest calibration error in out of the 11 experiments. Further, observe that many cases temperature scaling on top of AdaFocal does not offer any improvement (optimal temperature = 1). For the rest, the optimal temperature is close to indicating that AdaFocal produces innately calibrated models during training itself. The consistency of AdaFocal across other calibration metries is shown through AdaECE and classwise-ECE in Appendix F ECEaebias (15 and 30 bins), ECE -sweep (equal-width), and ECEEM-sweey (equal-mass) reported in Appendix G Significance of the results is confirmed through ECE error bars with mean and standard deviations computed over 5 runs in Appendix H
is able
Number of bins The ECE metries in the paper are reported using 15 bins. For AdaFocal training we experiment with 5, 10, 15, 20, 30, and 50 equal-mass (adaptive) binning when drawing calibration statisties form the validation range 10 to 20. Performance degrades when the number of bins are too small (<c 10) or too large (> 20), therefore, for the AdaFocal training the paper we use 15 bins well.
reported in Appendix | We find the best results to be from the
Dut-of-Distributtio (OOD) detection. Following Mukhoti et al. 2020). we report the perfor- mance of AdaFocal on an OOD detection task. We train ResNet-110 and RedeeResNNe26-111 10 on
While reproducing the baseline experiments in Mukhoti et al. 2020 obtained very similar results, therefore, we simply berrow the exact values maintain consistent comparison.
8

---

Under review as a conference paper at ICLR 2022
Datawl
Model
Cross
Entrogy Post
Beire Lass
MMCR
AdaFocail
T
Pre T
T
T
Post T
Pre
Post
Pro T
Post
KeaNcs-01
4.35
4.55
1.1522.0)
2556
1.700.91
1.55
ResNes-110 110
1.042.8)
1.25(1.2)
1.4222.8)
2,09
2.091.0)
(.7(11
0,65 L.6)
26
3.23
092/2.2)
1.251.00
3.29
0.866222)
4.36
L8400.)
1.56
084(08)
0.70100)
DesseNer-13 121
1.31(244)
1.531.00
5.1
1,612.5)
1.88
1.830.9)
1.23
1,220.0
(U66C1002)
ReiNes-01
17332
1422111
6.52
3641111
15.32
2.38(138
7.81
4001111
45
2
13
13100
CIEAR 100
Wide-Re Re---R---t--6-11
RecNes- 10
15333
19:05
2.88.222)
4.3.2.5)
4.31
7.88
4,651.2
2.701.1)
13.17
19.14
4.37(1 9)
3.8662.3)
11.00
4.84
5.891.1)
1,84(()
8.56
3.03
4.120120
1.6(1.1)
1.90
13
1 190110
1.31.))
DenseNet 12)
20.98
4,292.3)
5.17
2,291.1)
19.33
3.082.11
12.89
7,52/12)
3.73
1.310.1)
1.74
1,74000)
ImageNet
50
15.30
5.4144)
4,44
4.130.99
1300
5.500.
1523
6310077
1.76
12601
2.41
2:210.967
ImageNet
Newogrops
Global pool
17.92
2.39344
3.222.1
6.78222)
4.79
23RLD
6.92
2.191.5)
2.670.12)
Table I: Test set ECE(%) for different methods (pre and post temperature scaling). Underlined values mark the lowest error among Pre-T results and bold marks the overall lowest in the row.
Optimal temperatures are given in brackets. For AdaFocal, the values are averaged over 5 runs.
Dataset
Model
Cross Entropy
Brier Loss
MMCE
LS-0.05
FLSD-53
AdaFocal
ResNet-50
4.95
5.0
4.99
5.29
4.98
5.30
ResNet-110
4.89
5.48
5.4
5.52
5.42
5.27
CIFAR 10
Wide RedeeReeNNet26-11
3,86
4.08
3.91
4.2
4,01
4.5
DenseNet-121
5.0
5.11
5.41
5.09
5,46
5.2
RexNet
23.39
23.22
22.60
Res
22.73
25.1
23.07
23.43
22.51
22.79
CIFAR-100
Wide-ResNet-26-IB
20.7
20.59
20.73
21.19
20.11
20.07
DenseNet-121
24.52
23.75
24.0
24.05
22.67
22.22
Imygeme
Resnet-50
49.81
33.2
51,31
47.12
49.06
48.26
ImageNet
Resuct-50
27.08
28.53
27.22
20 Newsgroups
Global-pool CNN
26.68
27.06
27.23
26.03
27.98
28.53
Table 2: Test set error (%). The model with the lowest error is marked in bold.
CIFAR-10 10 as the in-distribution data and test on SVHN Netzer et al. (201T) and CIFAR-10-C Hendrycks & Dietterich (2019) (with level 5 Gaussian noise corruption) OOD data. Using entropy of the softmax as the measure of uncertainty. the corresponding ROC plots are shown in Figure 6 and AUROC scores are reported in Table T0 in Appendix ! We that models trained with AdaFocal outperform focal loss = 3 (FL-3) and FLSD-53, For the exact AUROC scores, please refer to Appendix J These results further highlight the benefit of inherently calibrated model produced using AdaFocal as post-hoc techniques such as temperature scaling, as shown in the figure, ineffective under distributional shift Snoek al. (2019).
1.0
10
FL3, Fre
3, Pre
ITA
3. Pre
FL Pre
0.4
ILSD-53, Pre
H-d, Posk
04
H-d, Pos.
FLSD-53. Pre
H-d, Pos.
FL5D-33 P
044
Pud. Post
FL5D-53, Fre
FL5D-53. Post
0.2
FLSD Post
62
FL5D-53. Post
FLSD 53, Post
AdoFoccl
AdaFocal
AdaFocal
AddFocal
0.
03
26
6o
1.0
3.0
62
64
16
co
10
30
62
64
16
60
62
64
os
aa
1.
FPR
SPRR
(a) SVHN: ResNct-110, WideResNet
(b) CIFAR- ResNet-110, WideResNet
Figure 6: ROC for ResNet-110 and Wide-ResNet- 26-Net226-11 10 trained on in-distribution CIFAR- 10 and tested on out-of-distribution (a) SVHN and (b) CIFAR- FL-3 refers Focal loss 7 = 3, Pre/Post T refers to before and after temperature scaling respectively.
CONCLUSION
In this work, we first revisit the calibration properties of regular focal loss and highlight the downside of using a fixed y for all samples. Particularly, by studying the calibration behaviour of different samples in different probability region, we find that there's no single 7 that achieves the best calibration over the entire region. We use this observation to motivate the selection of y independently for each sample (or group of samples) based on the knowledge of network's under//ovrr-cooffdeeee. We propose calibration- aware adaptive focal loss called AdaFocal that accounts for such information and updates the " at every step based on 7 from the previous step and the magnitude of network's inder//ver-confidencc. We find AdaFocal to perform consistently better across different dataset- model pairs producing innately calibrated models that most times do not substantially benefit from post-hoc processing of temperature scaling. Additionally, we find models trained with AdaFocal to be significantly better in out-of -ftt---disstibution detection task.

---

Under review as a conference paper at ICLR 2022
Reproducibility For reproducibility, we have include in the supplementary material zip file that contains the code base for running the experiments. For running particular experiments
CIFAR-10, ResNet-50, Cross entropy: python train.py daataset cifar10 -model resner50
-lsss cross_entrpee -num_bins 15 -e 400 -save-path Siteriieensccffarlo_reseeEE_
CIFAR-100, ResNet-50, Cross entropy: python truin.py -datasei cifur -model resnet50 -loss cross_ pross_enrropy -num bins 15 -e 400 -save-path experiments/cifar100 1etir_ereeeeSS____ _ce
Tiny-ImageNet, ResNet-50,, Cross entropy: python train.py -dataset tiny imagenet -model resnet50 ti -loss cross cross_entree _essreprrpp -num bins 15 -furst-milestone 40 -second-
milestone 60 100 -h 64 -th 64 -dataset-root data/tiny perioeenstinyymngeeeeeeeeeen 1aatianggeeeeeeett00_11111
-save-path
20 Newgroups, CNN, Cross entropy: python main.py -loss cross_ entropy -num-epochs 50 -num-bins 15 -save-path experiments/cmm ce.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 1950.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Puttern Recognition, 2009.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2017.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In International Conference on Learning Reprexentations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, 2019.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, volume 32, 2019.
Aviral Kumar. 20 newsgroups mmce. http:://gghhaab..ommeeerraaaaaae 2018.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2018
Ken Lang. Newsweeder: Learning to filter netnews. In in Proceedings of the 12th International Machine Learning Conference (ML95, 1995.
M. Lin, Q Chen, and S. Yan. Network in network. CoRR, 1312.4400, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
Jishnu Mukhoti. Focal calibration.
Bttees//gEaab..cm/ccrrrvssooooo
cal calibration] 2020.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip HS Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, 2020.
Rafael Müller, Simon Korbliihh, and Geoffrey E Hinton. When does label smoothing help? In
Advances in Neural Information Processing Systems, 2019.
11

---

Under review as a conference paper at ICLR 2022
Mahdi Pakdaman Nacini and Gregory F Cooper. Binary classifier calibration using ensemble of near isotonic regression models. Data Mining (ICDM), 2016 IEEE 16th International Conference on, 2016.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'1,, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2015.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word
representation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP). Association for Computational Linguistics, 2014.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, 1999.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration error estimation, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model's 's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In Proceedings of the Twenty-Secome International Conference on Artificial Intelligence and Statistics, volume 89, pp. 3459-3467. PMLR, 16-18 Apr 2019.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICMI, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In KDD, 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
12

---

Under review as a conference paper at ICLR 2022
APPENDICES
A ADAFOCAL'S GENERALIZATION TO LARGE SCALE DATASET (IMAGENET)
For ImageNet, FLSD-53 seems to perform very poorly in terms of calibration. The reason that due to higher values of y 5, 3 FLSD-53 becomes extremely under-confident in each bin leading a high calibration error. AdaFocal, on the hand, remains well calibrated which confirms that during training, unlike CE or FLSD-53, AdaFocal being aware of the network's current under/over- confidence (through the validation set) able to adjusts the 7s calibrated model every step. Further, in Figure 88 note the dynamics/evolution of in different bins. For ImageNet, we find AdaFocal's y - 0 which makes sense because, from Figure 7 cross entropy (i.c. 0 for every bin) is much better calibrated than FLSD-53 and AdaFocal (starting from 7 = 1) settles down as CE (7 = 0). Note that for an unseen dataset-model pair is not possible to know beforehand whether CE focal loss will perform better. However, from these experiments, we find strong evidence that, for any dataset-model pair, AdaFocal will lead to the ys that result in the best calibration.
way that maintains a well
45
CE
15
FLSD-53
40
Error (%)
AdaFocal
ECE (%)
35
5
30
25
0
20
40
60
80
20
40
60
80
Epoch
Epoch
Figure 7: Test set error and calibration of ResNet-50 trained ImageNet. Left: Error, Right: ECE.
13

---

Under review as a conference paper at ICLR 2022
CE
FLSD-53
AdaFocal
Bin 14
Bin 12
Bin 10
Bin 8
C,al -vel
-0.1
-0.1
-0.1
-0.1
20
40
00
20
40
60
80
40
00
0
20
40
60
00
Artaara
1.0
1.0
MbFacc
1.0
1.0
0.5
0.0
Bin bourdariiss
1.00
20
40
60
1.0
40
60
80
1.0
40
60
80
1.0
20
60
80
0.95
0.5
0.5
0.90
40
40
60
6
4
0.2
0.2
0.2
0.2
Cuil Sui--Ay Av
0.0
0.0
0.0
0.0
00.2
-0.2
0.5
0.5
0.5
0.0
0.0
0.0
0.0
Bin toundares
0.0
00
0.0
0.0
60
0.0
60
Epoch
Epoch
Epoch
Epoch
Figure 8: Dynamics of y and calibration in different bins when ResNet-50 is trained on ImageNet. Each bin has three subplots: top: Eval, Cval,t Aval, middle: evolution of Yr and bottom: bin boundaries. Black dotted line in top plot represent zero calibration error. We observe that for most of the bins AdaFocal approaches 7 (which is the CE entropy loss) that result the best calibration.
14

---

Under review as a conference paper at ICLR 2022
B CALIBRATION BEHAVIOUR OF FOCAL LOSS IN DIFFERENT BINS
In the main paper, we showed the calibration behavior of different focal losses for ResNer50 trained on CIFAR-10 for only a few bins. For completeness, the rest of the bins and their calibration error
E, Caat,i A are shown in Figure 9 for focal losses with 0,3, 4, 5. We observe that there's no single y that performs the best across all the bins. Rather, every bin seems have particular that achieves the best calibration.
Bi monurrreeee
B0 ussundaran -h
Cl
y-
Figure 9; ResNet50 model trained on CIFAR-10 using different focal losses 0,3,4,5, Top: Eval, Cral,i Aval, Bottom: bin boundaries. The statistics are computed on the validations set (5, 000 examples) using 15 equal-mass bins. The black horizontal line in top subgfigure represents zero calibration error Evai, 0.
15

---

Under review as a conference paper at ICLR 2022
C CORRESPONDENCE BETWEEN CONFIDENCE OF TRAINING AND VALIDATION SAMPLES
Dme
Cen
Soundures contileecs
Figure 10: ResNet50 trained on CIFAR-10 with focal loss 7 = 3. It shows that Ctrain,rrue and
Cop are almost the same during major part of the training. This is because the model approaches towards 100% accuracy on the training set, the top predicted class and the true class for training sample become the same.
16

---

Under review as a conference paper at ICLR 2022
TARRO DRATN DAAEET TAA
Teams we-.ER--T Cerceee
Tooal
Tocal
Traing S cetmmben
Thaniin Teeeae bend
Than and Sereemmmoo
Thaing 202. catenbin
food
lsce Louing-
Rcdd p- 2409g-
Sertyy veldeoo 2o
Focal vehadon
Figure 1I: Independent binning: training samples and validation samples are grouped indepen- dently into training- bin and validation bin respectively. The top subfigure for each bin shows the correspondence between average confidence of a group of training samples Ctrain,true, and group of validation samples Ceal.top.i when ResNet-50 is trained on CIFAR-10 with focal loss y 0, 3. 5. The binning is adaptive with 15 equal-mass bins. Solid line: Ctrain,true, in training-bin i, Dashed line: Cval, top.i and Star-dashed line: A in validation-bin bin i. The bottom subfgure shows the bin binudary only for focal loss y = 3.
17

---

Under review as a conference paper at ICLR 2022
Cir-E Tooal
Tood
Tocal
Foca p- 2.wlaaao--4
2
Weicer 2nd
Saiatte berd
Rood
Vi Caltor Deyt
0 food
Tocal
Figure 12: Common binning: training samples are grouped using the bin boundaries of the validation-
bins. The top subfigure for each bin shows the correspondence between average confidence of group of training samples Ctrain,true, and group of validation samples Caat,top. when ResNet-50
trained on CIFAR-10 with focal loss = 0,3. 5. The binning is adaptive with 15 equal-mass bins. Solid line: Ctrain.trae. in validation-bin i, Dashed line: Cool,top,i and Star-dashed line: A in
validation-bin i. The bottom subfgure shows the bin binudary only for focal loss 3.
18

---

Under review as a conference paper at ICLR 2022
D CALFOCAL LOSS
200
13 F00 10
5p207
Figure 13: ResNet-50 trained on CIFAR-IO using CalFocal loss. Eap. CalFooaal loss in Eq. of the main paper. Legend À = CalFocal loss in Eq. 2 with common 7%6 for all training samples in bin b. Ce refers to cross entropy.
E DATASETS AND EXPERIMENTS
E. DATASET DESCRIPTION
CIFAR-10 Krizhevsky (2009) This dataset contains 60, 000 coloured images of size 32 32, which are equally divided into 10 classes. A split of 45, 000/5, 000/10, 000 images is used as train/validation/testt respectively.
CIFAR- Krizhevsky (2009): This dataset contains 60, 000 coloured images of size 32 x 32, which equally divided into 100 classes. A split of 45, 000/5, 000/10, 000 images is used as train/validation/tess sets respectively.
ImageNet Russakovsky et al. (2015): ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012-2017 is an image classification and localization dataset. This dataset spans 1000 object classes and contains 1,281,167 training images and 50,000 validation images.
Tiny-ImageNeel Deng et al. (2009): It is subset of the ImageNet dataset with 64 x 64 dimensional images and 200 classes. It has 500 images per class in the training set and 50 images per class in the validation set.
20 Newsgroups Lang 1995): This dataset contains 20, 000 news articles, categorised evenly
into 20 different newsgroups. Some of the newsgroups are very closely related to each other (e.g. comp.sys.ibme ibm. pe. hardware comp.sys.man wws..maa.aardwarr........... while others are highly unrelated (e.g forsafo / ments.
We use train/validationntest split of 15, 098 098/900/3, 999 docu-
E.2 EXPERIMENTAL DETAILS
For all our experiments, we have used Nvidia Titan X Pascal GPU with 12GB memory.
CIFAR-10 and CIFAR-100: We use SGD with momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the first 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs. We use a training batch size of 128. The training data is augmented by applying random crops and random horizontal flips.
Imag-Im We use SGD with momentum of 0.9 as our optimiser, and train the models for 100 epochs with a learning rate of 0.1 for the first 40 epochs, 0.01 for the next 20 epochs and 0.001 for the last 40 epochs. We use training batch size of 64. Note that we use 50 samples per class (i.e. a total of 10000 samples) from the training set as the validation set. Hence, the training is only on 90000 images. We use the Tiny- ImageNet validation set
test set.
ImageNet: We use SGD as our optimiser with momentum of 0.9 and weight decay 10-4, and train the models for 90 epochs with learning rate of 0.01 for the first 30 epochs, 0.001 for the next 30 epochs and 0.0001 for the last 30 epochs. We use a training batch size 128. We divide the 50,000 validation images into validation and test set of 25,000 images each.
20 Newsgroups: We train the Global Pooling CNN Network Lin et al. (014) using the Adam optimiser, with learning rate 0.001, and default betas 0.9 and 0.999. We used Glove word embeddings
19

---

Under review as a conference paper at ICLR 2022
Pennington et al. (2014) train the network. We train the model for 50 epochs and use the model at the end to evaluate the performance.
All our experiments mentioned above are set to their default values. For CIFAR- 10/AR00 and Tiny-Imagen ImageNet, AdaFocal is implemented top of the base code available from Mukhoti (2020). The code for 20 Newsgroups is implemented in PyTorch by adapting the code (TensorFlow) available from Kumar (2018).
implemented in PyTorch. The hyperparameters that not explicitly
The experimental results in the paper reported for the model at the end of (1) CIFAR- 10/100: 350 epochs, (2) ImageNna::Net: 100 epochs, (3) ImageNet: 90 epochs, and (4) 20 NewsGroups: 50 epochs.
F ADAECE AND CLASSWISE-ECY ECE PERFORMANCE
Here, we compare the performance of AdaFocal against the baseline methods in terms of AdaECE and classwise-Fi ECE in Table and Fudd4rees respectively. For CIFAR-10/100, the values are reported for the model at the end of 350 epochs: for Int-een,,,,,, at the end of 100 epochs; and for 20 NewsGroup dataset, the end of 50 epochs. From these tables, we observe that AdaFocal outperforms all the baseline methods by substantial margin, especially if we compare the pre-temperature scaling results.
Dataset
Model
Cross Entrups
Brier Loss
MMCE
LS-0.8
FLSD--3
AdaFural
Pre'T
Post T
Pre
Post
Pre'T
Post
Pre.T
Post
Pre
Post
PreTT
PostT
ResNet 50
4.33
2.142255
1.74
(.23(11)
4.55
2.16/2.0)
1.89
29220991
1.50
1.26(111)
0.56
0.880.9)
CIFAR-10
Wide ResNee 26-1026-10 10
ResNer-110 110
3,23
4,4
1.6(2.2)
1,99228)
2.6
1.7
1.7(1.2)
1.7(1)
3.29
5.06
2.52(2.8)
1.6(2.)
4.44
4.27
2.44.0.8)
4.44(1)
2,07
1.52
1.6(1.11
1.3800.9)
0.44
0.64
0.42(1.))
0.44(1.0)
DenseNet 121
4,51
2.13/2.4)
2.03
200(1)
5.1
2.29/2.5)
3,33(09)
1.42(1)
0.5411.0)
1.4222.1)
6.2
3.4(1.)
2.38(18)
1.721.00
CIFAR-100
Wide Roo-RRoNNee2--ll
RosNet-110
15:35
19.05
5.862.33
2.89(2.2)
7.33
4.22
4.59(1.2)
2.8((.1)
19.14
13.16
4.5((19)
435/2.3)
11.12
8.501.3)
5.1(1)
2:75
3.961.2)
1.630.1)
2.22
1.57(100)
2.22(1 0)
DenseNeet 121
20.98
5.09/2.3)
5.04
2,56(1.3)
19.13
3,07/2.1)
12,83
8.92(1.2)
3.55
1.24(1.1)
1,54
1.59(0.)
Iny ImagoNei
ResNet 50
15:23
3.4
4.37
4,000091
130
3.36(1.3)
15.28
6.25K071
1.42
1,400)
1.26
1,261.)
ImageNet
ResNett-0 30
201
1,500.9
16.77
2.820075
3.75
0,84009)
20 Newsgroups
Global CNN
17.91
2.331.4)
13.57
3.11(23)
15.21
6.47/2.2)
2.6(1.3)
2.3511.5)
2.38(1.)
Table 3: AdaECE(%). Underline marks the lowest error among pre-tempera scaling values.
Bold marks the overall lowest in the row.
Model
Kniropy
ESaLas
FLSDE3
Post
Pre
Post
T
ReINer:30
031
0.5(2.3
0.46
0.3/1.11
0322210)
0221.1)
0.321099
CIFAR-10
Widt-ResNin-26-100
ResNet- 110
10
0.68
0.91
0.3712.2)
0.50225)
0.44
0.39
0.5011.2
0.441)
0.70
1.04
0.35/2.2)
0.55228)
0.80
0.66
045(08)
0.66111
0.48
041
0.44(1.1)
0.31(09)
0.27
0.33
0.28(1.)
0.33/1.0
DenseNe:-12
0.92
0.472244
0.16
1)
1.04
0,570.5)
0.60
0.50(0.)
0.41
044(1)
0.30
0.30(L.))
3o=Nei:0
0.38
0.222.11 11
022
0,3R1.1)
0.3
022(1
0.25
021(1.1)
0.201.1)
IL.00
0.200.0
CIBAR-100
Wate-RcsNei-26-10
ResNet: 110
0.41
0.34
0.2022.)
021(2.3)
0.24
0,191111
0,231.2)
0.42
0.0(1.9)
0.2212.5)
0.26
0.22(.1)
0.16(1.)
0.21112)
0.19
0.19(1.0)
0.19(1.0)
02222.3
0.30
0.21(11)
0.42
02242.)
2)
0.201.. 1.D
0.201.0)
Tin- ImapeNNr
KooNen:00
0.23
6.167444
0.16
0.1600.))
0.21
0.21
011007)
0.660)
0.16
0.16/100
ImageNer
KcsNen-1
8.03
003
0.000071
R003
20 Newsgruups Gi pool CNN
1.95
0.333341
136
0.2/2.3)
1.77
L1002.)
099
0901.1
1410
L
1.02
1.021.07
Table 4: CCE)))) Underline marks the lowest error among pre-temperature scaling values. Bold marks the overall lowest in the row
20

---

Under review as a conference paper at ICLR 2022
G
DEBISED ESTIMATES OF ECE
Dataset
Model
Cross Entropy
FLSD-53
AdaFocal
Pre
Post T
Pre T
Post T
Pre T
Post T
ResNet-50
4.05
1.7(2.5)
1.62
1.62(1)
0.47
0.82(0.9)
ResNet-110
4.38
2.2(2.7)
1.82
1.3(1.1)
0.32
0.32(1)
CIFAR-10
Wide-ResNet-26-11 10
3.52
1.89(2.2)
2.01
1.5(0.9)
0.59
0.25(1.1)
DenseNet-121
4.26
2.15(2.3)
1.56
1.93(0.9)
0.42
0.42(1)
ResNet-50
17.73
3.86(2.2)
5.32
2.92(1.1)
1.46
1.46(1)
ResNet-110
19.44
6.01(2.3)
7.31
3,55(1.2)
1.35
1.35(1)
CIFAR-100
Wide-ResNet22611 10
14.91
3.32(2.1)
2.53
2.53(1)
2.12
2.12(1)
DenseNet-121
19.82
3,44(2.3)
2.29
2.12(1.1)
1.27
1.27(1)
ImageNete
ResNet-50
16.16
5,44(1.5)
2.00
2.00(1)
0.84
0.84 (1)
ImageNet
ResNet-50
2.89
1.42(0.9)
16.76
2.58(0.7)
3.7
0.61(0.9)
20 Newsgroups
Global-pool CNN
18.36
5.23(4.1)
8.94
0.94(1.)
1.84
1.84())
Table E.Faia (%) 15 bins. Underline marks the lowest error among pre-temperature scaling values. Bold marks the overall lowest in the row. Optimal temperature is selected based on the lowest ECE on the validation set.
Dataset
Model
Cross
Entropy
FLSD-53
AdaFocal
Pre T
Post T
Pre T
Post T
Pre T
Post T
ResNet-50
4.03
1.73(2.5)
1.57
I,57(I)
0.46
0.93(0.9)
ResNet-110
4.38
2.21(2.7)
1.81
1.2((1.))
0.26
0,26(1)
CIFAR-10
Wide-ResNee-26-1i1 10
3.5
1.84(2.2)
1.98
1.5(0.9)
0.66
0.24(1.1)
DenseNet-121
4.24
2.16(2.3)
1.95(0.9)
0.64
0.64(1)
ResNet-50
17.71
3.66(2.2)
5.52
2.99(1.1)
1.55
I.55(1)
ResNet-1
19.43
6.31(2.3)
7.23
3.86(1.2)
1.17
1.17(1)
CIFAR-100
Wide-ResNeet26-10
14.9
3.4(2.1)
2.46
2.46(1)
2.12
2.12(1)
DenseNet-121
19.81
3,49(2.3)
2.18
2.31(1.1)
1.08
1.08(1)
Tiny-ImageNet
ResNet-50
16.11
5.5(1.5)
1.81
1.81(1)
0.64
0.64(1)
ImageNet
ResNet-50
2.84
1.39(0.9)
16.76
2.55(0.7)
3.67
0.91(0.9)
Newsgroups
Global-pool CNN
18.39
5.51(4.1)
8.98
1.89(1.6)
1.92
1.92(1)
Table 6: ECEdebias (%) 30bins. Underline marks the lowest error among pre-temperature scaling values. Bold marks the overall lowest in the row. Optimal temperature is selected based on the lowest ECE on the validation set.
Dataset
Model
Cross
Entropy
FLSD-53
AdaFocal
Pre
Post T
Pre T
Post T
Pre T
Post T
ResNet-50
4.05
1.17(2.5)
1.31
1.31(1)
0.48
0.92(0.9)
ResNet-110 10
4.38
1.34(2.7)
1.96
1.05(1.1)
0.5
0.5(1)
CIFAR-10
Wide-ResNet-26-10
3.53
1.14(2.2)
1.22
0.92(0.9)
0.64
0.36(1.1)
DenseNct-121
4.26
1.43(2.3)
0.94
1.9(0.9)
0.33
0.33(1)
ResNet-50
17.72
2.81(2.2)
5.54
2.17(1.1)
1.7
I.7(1)
ResNet-110
19.44
3.74(2.3)
7.36
3.86(1.2)
1.54
1.54(1)
CIFAR-IO0 100
Wide-ReeNNet22-110
14.99
3.11(2.1)
2.69
2.69(1)
2.02
2.02(1)
DenseNet-121
19.82
2.58(2.3)
2.24
2.34(1.1)
1.27
1.27(1)
Tiny-ImageNe
ResNet-50
16.19
5.56(1.5)
2.14
2.14(1)
1.37
1.37())
NmageNet
ResNet-50
2.84
1.62(0.9)
16.77
2.52(0.7)
3.68
1.1(0.9)
20 Newsgroups Global-pool CNN
18.39
5.14(4.1)
9.03
1.48(1.)
2.06
2.06(1)
Table 7: ECEEW sweepp (%). Underline marks the lowest among pre-temperature scaling values. Bold marks the overall lowest in the row. Optimal temperature is selected based on the lowest ECE on the validation set.
21

---

Under review as a conference paper at ICLR 2022
Dataset
Model
Cross Entropy
FLSD-53
AdaFocal
Pre T
Post T
Pre T
Post T
Pre T
Post T
ResNet-50
4.05
1.43(2.5)
1.54
1.54(1)
0.04
0.7(0.9)
ResNct-110
4.38
1.34(2.7)
1.83
1.32(1.1)
0.4
0,4(1)
CIFAR-10
Wide-ResNet-26-11
3.53
1.41(2.2)
1.64
1.55(0.9)
0.38
0.32(1.1)
DenseNct-121
4.27
2.17(2.3)
1.58
1.98(0.9)
0.34
0.34(1)
ResNet-50
17.72
0.51(2.2)
5.51
2.36(1.1)
1.89
1.89(1)
RcsNct-110
19.44
3.71(2.3)
7.34
3.65(1.2)
1.58
1.58(1)
CIFAR-100
Wide-ResNet-26-10
14.92
2.62(2.1)
2.62
2.62(1)
2.25
2.25(1)
DenseNct-121
19.82
3.12(2.3)
2.25
2.31(1.1)
1.47
1,47(1)
Tiny-ImageNe
ResNet-50
16.19
5.44(1.5)
2.10
2.10(1)
1.6(1)
ImageNet
ResNet-50
2.93
1.63(0.9)
16.77
2.58(0.7)
3.73
0.92(0.9)
Newsgroups
Global-pool CNN
18.38
5.53(4.1)
8.95
2.13(1.6)
2.22
2.22(1)
Table 8: ECEEM- (%). Underline marks the lowest emror among pre-temperature scaling values. Bold marks the overall lowest in the row. Optimal temperature is selected based on the lowest ECE on the validation set.
22

---

Under review as a conference paper at ICLR 2022
H ECE ERROR BARS
Cr96 15, Rendtrr 3D
CPAR Mrab Render
CHAR 100, Resten 20
CINAR 125, Uidee Rectlet
torgefes, Rected
Figure 14: ECE error bars with mean and standard deviation computed over 5 runs with differ- ent initialization seed. The dark and light colors show pre and post temperature scaling results respectively.
I NUMBER OF BINS USED DURING ADAFOCAL TRAINING
Experiment details: CIFAR-10, ResNet50 trained for 350 epochs. The reported results below without temperature scaling. Our method AdaFocal with 5, 10, 15, 20, 30, and 50 adaptive (equal mass) bins FLSSD--3.. Note here that there two types of binning:
During training: the binning that is performed on the validation set from where AdaFocal draws calibration related information to adjust the y. This corresponds to the columns the table below.
During evaluation: once we have trained model, the binning that is used to compute different ECE metrics. These correspond to the rows in the table below.
FLSD-33
Sbins
1Obins
15bins
20bins
30bins
50bins
ECE( 5bins)
1.35
0.76
0.82
1.16
AdaECE(15 (15bins)
1.67
0.63
0.53
0.56
0.4
0.84
1.1
ECE dehins (15bins)
1.62
0.5
0.44
0.47
0.25
0.79
1.07
ECE debias(30bins)
1.57
0.73
0.43
0.46
0.27
0.72
1.06
BW BW_ ECE _CE_sweep
1.31
0.66
0.43
0.48
0.48
0.8
1.08
EM RM_ECE_sweep
1.54
0.53
0.38
0.07
1.08
Table Effect of number of bins used for AdaFocal.
AUROC FOR OUT-OF-DISTRIBUTTN DETECTION
For ResNet1 10 on CIFAR-10/SVHN, we were not able to reproduce the reported results 96.74, 96.92 for FL-33 in Mukhoti et al. (2020). Instead we found those values to be 90.27, 90.39 and report them in IO
Dulaset
Model
Cross Entropy
Brier Loss
MMCE
LS-8.08
FL.3
FLSD--3
AdaForal
Pie
Posi
Pie
PoaT
Pre T
Post
Pir
Post
Pte
Post
Pre.T
PostT
Pre
CIFAR-IO USVEN
RosNer- TT
9682
61.71
97.62
39.46
94.51
14.30)
95.15
94.51
85.31
97.35
R3.90
97.95
68.88
84.63
53.06
$4.66
93127
90.92
90..0)
91.30
93. 08
181.35
001.00
93.11
96,19)
96.63
CIFAR CIPAR-IBC
ReiNet
31.6
77.53
75.06
30.88
$478
85.00
85.03
83.86
82.
71.56
8172
70.02
71.10
72.17
71,16
72.18
80111
83.33
79.78
84.00
82.06
80.00
$1.38
30.76
$4.96
89.52
Table 10: AUROC (%) of models trained on CIFAR-10 as the in-distribution data and tested on SVHN and CIFAR-10-C out-of-distribution data.
K
MOVING AVERAGE -UPPAAE RULE
For the focal loss in the paper C(pn. -(1 (1-P)) pm. AdaFocal is given by
log the unconstrained 7-update rule for
M++ = " * - Avat,t1))
(5)
" exp( Eval,t+))
(6)
23

---

Under review as a conference paper at ICLR 2022
If instead we exponential moving average update Y, then the update rule (let's call it MA-a) is given by
7++1 (7)) (e (eattttt)) 1-a
= - & t Fvald & ((a,,))
- * gllabaass(--a)aullle
10
5
CE
AdaFocal
8
MA-0.05
Error (%)
ECE (%)
MA-0.1
MA-0.2
2
MA-0.5
MA-0.8
20
100
200
300
100
200
300
Epoch
Epoch
(a) Error (%)
(b) ECE (%)
Figure 15: ResNet-50 trained on CIFAR-10 10 with cross entropy (CE), AdaFocal, and the moving average 7-update rule (MA-a). We see that MA-a is not effective as AdaFocal's 7 update rule.
The evolution dynamics of b is given in Figure 16
24

---

Under review as a conference paper at ICLR 2022
AdaFocal
MA-0.05
MA-0.5
MA-0.8
Bin 14
Bin 12
Bin 9
0.01
0.01
0.01
Cyai Avai
0.00
0.00
0.00
-0.02
0.02
-0.02
100
200
300
100
200
300
100
200
300
1.2
1.10
1.10
Vt
1.05
1.05
1.0
1.00
1.00
0.9
100
200
300
100
200
300
100
200
300
Bin
Bin
Bin 0
0.10
0.10
0.10
Ayal
0.05
0.05
0.05
0.00
0.00
0.00
0.05
0.05
0.05
0.10
100
200
300
100
200
300
100
200
300
20
1.2
10
10
1.0
0.8
100
200
300
100
200
300
100
200
300
Epoch
Epoch
Epoch
Figure 16: Training ResNet-50 on CIFAR-I0 using AdaFOOal and MA-a. Each bin has subplots: top:
Eval. = Ceat.; - Avat, bottom: evolution of 7t. Black dotted line in top plot represent zero calibration error. We observe that for MA-a the y for different bins are not as free move around as that under AdaFocal.
L MULTIPLE RUNS OF ADAFOCAL WITH DIFFERENT Tmax
Due to the stochastic nature of the experiments, AdaFocal 7s may end up following different trajectories across different runs (initialization), which in turn might lead variations the final
results. In this section, we look the extent of such variations for (1) unconstrained (2) capped by Tmuax 20 and (3) y capped by Tmax = 50. We study this for ResNet-50 trained multiple times on CIFAR-10 starting with different random seed.
L. ADAFOCAL, 2max 20
In Figure 17 we observe that AdaFocal with Ymax = 20 is consistently (9 out times) better than FLSDD-3. Figure shows the evolution of across different runs.
25

---

Under review as a conference paper at ICLR 2022
5
10
8
4
Error (%)
ECE (%)
6
4
CE
FLSD-53
2
AdaFocal
0
0
100
200
300
100
200
300
Epoch
Epoch
(a) Error %)
(b) ECE (%)
Figure 17: ResNet-50 trained on CIFAR-10 using 5 runs of cross entropy (CE), FLSD-53 and AdaFocal with different initialization seed. AdaFocal with "mmxx 20 is consistently better.
Bin
Bin
14
28
10
20.0
37.5
15.0
50
300
120
200
250
300
150
Figure 18: Evolution of for 9 runs of ResNet-50 trained on CIFAR-10 10 using AdaFocal Tmax 20.
L.2 ADAFOCAL, 7max 50
In Figure 19 we observe that AdaFocal with Ymax
50 has more variability than AdaFocal
Tmax 20 but is mostly better than FLSD-53. Figure 19 shows the evolution of 7 across different
runs.
crees entrady
Test set ECEES)
Cress irtreg
Test set AdvECEIS)
Crow entigg/
10
Test casswiSe ECENN)
FLDD 33 Heol
SO8, Fm 50 nrai
210
115
Adghoua
170
615
(a) Error(S) %
(b) ECE (%)
(c) AdaECE (%)
(d) classwise-ECE(())
Figure 19: Plots for ResNet-50 trained CIFAR-10 using entropy (1 run), FLSD-53 (3 runs) and AdaFocal (5 runs). AdaFocal with Ymax 50 although is mostly better than FLSD-53 does exhibit greater variability than Tmax
20.
26

---

Under review as a conference paper at ICLR 2022
Bin
Bin
Bin
1.3
Biri
Biri
Bin
14
10
30 3CD
130
300
250 300
250
Figure 20: Evolution of n for multiple runs of ResNet-50 trained on CIFAR-10 using AdaFocal
Tmax 50.
L.3 ADAFOCAL, UNCONSTRAINED
Y
In Figure 21 we observe that AdaFocal with unconstrained 7 does exhibit some variability across different runs: 7 out of 9 times it performs better than FLSD-53 whereas the other two times it is similar or slightly worse.
croes entropy R.50 55 rund Adef uncorataaieed
Cross entrary
Text set ECEESS
Test set AduECE(S)
entroyy
10
110
Test casowise ECENS)
310
110
613
(a) Error %)
(b) ECE (%
(c) AdaFCE (%)
(d) classwise-ECI((5
Figure 21: Plots for ResNet-50 trained CIFAR-10 using cross entropy (1 rum), FLSD-53 (3 runs) and AdaFocal (9 runs). AdaFocal with unconstrained does exhibit some variability across different runs: 7 out of times it is better than FLSD-53, two times it is similar slightly worse.
The above behaviour is mostly due to the variations in the trajectory of ys for lower bins, as shown in Figure 22 For higher bins, we see the ys settling to similar values, however for lower bins, as the ?s are unconstrained they blow up to very high values.
Bin 14
Bin 12
Bin 10
Bin 8
Bin
Bin
Bin 1
Bin
1750
1500
20/0
1020
730
900
Figure 22: Evolution of 7 for multiple runs of ResNet-50 trained on CIFAR-10 using unconstrained AdaFocal.
27

---

Under review as a conference paper at ICLR 2022
M ERROR, ECE AND BIN STATISTICS PLOTS FOR REST OF THE EXPERIMENTS
(50
ECE (%)
AduRCE (N))
FLSD-53
(2314
AdaFocal
PLS0-33
F.50-3
Esech
Epoch
Epoch
Epoch
Figure 23: ResNet-50 trained on CIFAR-I0 with cross entropy (CE), focal loss = 3, FLSD-53 and AdaFocal. (a) Error, (b) ECE, (c) AdaECE and (d) classwise-FCCF. AdaFocal achieves the lowest calibration error while maintaining similar enror performance.
28

---

Under review as a conference paper at ICLR 2022
Teat ser ErrantS
TOTT SCT ECEUSS
THE Set AdNESEIS)
TAST DET CEREEAAI ECEUSS
PE-D-VE atslarn
BON
0.01
Tpect
Touch
Bin 24
2m
Bin 10
FLSD.53
FISD-53 AdaFocal
FLSD-33
FLSD-53 Adefocal
Adafocal
AdaFocal
AtsFocal
PLSD
2416
Jal PLSD-53
PLSD Adofacal
FLSO 55
Adafocall
Adefacal
Adsfocal
Bi14
Sir
FLSD
FLSD 53
PLSO Adwfocal
FLSD 35 AdeFocal
Adaeocaal
Addefical
Allofozst
FLSD 5,
FLSD-55
PLSD
FLSD-53
AduPocal
200
Figure 24: CIFAR-10, ResNet-110: Test set Error, ECE, AdaECE, classwise-ECE and Validation set bin information used by AdaFocal during training.
Enrontss
AdMECERS)
ECE!N)
Cross Ertrppy
Cow
Entiony
Asshoca
C3TC -220
Bin
CCD
130
Mr
FLSD-53
FLSD-53 Adasacal
FLSDI53 AdaFacal
FLSO-53 AdaFocal
56
AdaFacal
Actevocal
102
PLSO $3 Adaf ocal
FLSD-55
PLSD 53
FLSD.53 Adafocca
Adat cocal
AdaFocal
Figure 25: CIFAR-10, Wide-ResNet. Test set Error, ECE, AdaECE, classwise-ECe and Validation set bin information used by AdaFocal during training.
29

---

Under review as a conference paper at ICLR 2022
Test Enrant'
set ECEESS
Tep net AdMESEVS)
Chseeatte ECELSS
PUNDAN
530
021
Tpeck
Bin 24
3m 12
Bm 20
Dire
FLSO-53 Adafocal
FLSD-53 Adde Scal
FLSD.53 Adafocal
FLSO.53
Ada Focal
130
P3 Adefocal
CA14
PLSD-33
PLSO 33 Adafocal
FLSO 53
Adal acal
Sir
BILO
PLSO
FLSD 33
PLSD AdaFaccel
FLSO 53
Adafoocal ocal
Adaf Scal
Adefocal
06 Lo0nlaee
FLSD 53
FLSD-35
FLSD
FLSO
AduPocal
220
Figure 26: CIFAR-10, DenseNett121:: Test set Error, ECE, AdaECE, Mlasswise-ECE and Validation set bin information used by AdaFocal during training.
Criss ACDECE!S)
20
ECE ING
Whr
700
Adstvaal
CE
FLSD-53
AdaFocal
20
Cpcen
Tpach
C3tt A200
atn
3in
dedsPoccal
FLSDISa
FLSD Adafacal
FLSD-53 Adiaacaa
FLSD-53
Adlaoocal
AdaFocal
E--cot-- 520
Bin
3in
ades-ocal
FLSDi53 Adal ocal
FL5D.53
FLSD-53 AdaFical
PLSD-53 AdeFocal
Adafocal
Figure 27: CIFAR-100, ResNet-50: Test set Error, ECE, AdaECE, classwise-ECE and Validation set bin information used by AdaFocal during training.
30

---

Under review as a conference paper at ICLR 2022
Text SCT ACOESEENI
TEAT diatewire ECEVS6
ECE (R)
75
Adehoca By
PUNDING
14
PL50
Crass EXEROO
Acol
%
Epoeh
Spacc
E--Cot moo
Dir
2tn 1-2
Bin
3m
FLSD:53
FLSD-53 Adafocal
FLSD-53 Adafacaa
FLSD-53 AdAFocal
Audia Focal
PLSD
FLSD:53 53
FLSD Adafacal
PLSD
Adal ocal
Bin
Bin
Zmn
FLSD
FLSD
FL3D Adafacal
"LSD Adieoocal
Pajal noa
Adafocal
BC Counsieee
PLSD-33
FLSD/33
LL3D-93
PLSD-33
AdoPocal
Figure 28: CIFAR-100, ResNet- 110: Test set Error, ECE, AdaECE, classwise- ECE and Validation set bin information used by AdaFocal during training.
ECE1S))
ACDECE!'S
Crew Berocyy
Crems Drirrpy
Crass
Alshoual
14.0
Agahatal
FIAnE PrEteey PENEIILE
C3CC -220
3in
FLSD-53 Adaaocaal
FLSD.53
FLSD-53 Adaaacaa
FLSD-53
AdaFocal
AdaFacal
3in
Adar-ocal
acalroccll
FLSD-13 Adal ocal
FL
FLSD-53 AdaFical
PLSD-53 AdeFocal
Adafocal
Figure 29: CIFAR-100, Wide-ResNef Test set Error, ECE, AdaECE, classwise-ECE and Validation set bin information used by AdaFocal during training.
31

---

Under review as a conference paper at ICLR 2022
Test ser Etran'st
Test ECEIN)
Text SEt AGOESEENI
Test ciassaire ECEV5)
PEND-VE atsfare
SENDINT
PERDONS
Crass ExErepy
Acol
Tpect
E--Cot- -oo
Dirt 24
Dtn 12
Din 20
3m
FLSD-53
FLSD. 53
FLSD-53 Adal Cocal
FLSO-52 Adafocal
Audal Focal
Adafaca!
FLSD-53
FLSD Adefocaa
FLSD Scal
PLSD
ocal
Bin
Bin
Zmn
FLSC 23
FLSD Adefacal
FLSD Adafiical
FLSD 53
Pddel Dical
AdeFocal
Adafocal
BC Counsire
PLSD-33 23
FLSD:53
FL3D-93 93
PLSD-33
AdoPocal
Figure 30: CIFAR-100, DenseNet- 121: Test set Error, ECE, AdaECE, classwise ECE and Validation set bin information used by AdaFocal during training.
ACDECE!S)
ECE(%)
Crase Ehtrops
Cras Datreey
SCE
760
Adshucal
40
100
300
Tpach
Bil
C3tt
561 199002011
FLSD-53
FLSD.sa Adaaacal
FLSD-52 Adaaocal
"LSDi3 Adafocal
Adiaoccal
Bin6
Din
Buidd ASIA A-Mbola
Aon Ad W NWW
DA 00//C0tee
FLSD-53 Jdatocal
FLSD-51
FLSD-53 Adal Tocal
"LSD.53 AdaFocal
Adal ocal
Figure 31: Tiny- ImageNe,,,, ResNet-50: Test set Error, ECE, AdaECE, classwise- -CC and Validation set bin information used by AdaFocal during training.
32

---

Under review as a conference paper at ICLR 2022
Test Set AIA
Teat SCT classnise- PNE
15
Ada edafocal
data Tacal
ECE
10
C2
RLSD 55
201
Epech
Towe
Birt 24
Sin 12
Bir 20
Sm
FLSD-53
FLSD.33 SAAA Adafocal
FLSD-S2 53
LLS--3 AdaFoca
Adlal nocal
AdaFocaal
80 3210000
PLSD-93
FLSD-55 53 Adefocal
FLSD-55 53
FLSD AdeFocal
oocal
Adal bcal
Bin
Bt1 4
Bin
PLSD 23
FLSD 33
FLSD Adaficaal
PLSD 33 AdeFocal
Adiel beal
Adefaca!
Adol
Adafscal
BC CouRSeee
PL3D-33
FLSD-33
FL3D:93 AdaFocal
PLSD-33 AdcFosal
AdoPocal
Figure 32: 20 Newsgroups, CNN: Test set Error, ECE, AdaECE,, classwise-ECE and Validation set bin information used by AdaFocal during training.
33


Reviews:
[{"review_id": "rQ2yLbceCka", "reviewer": "Reviewer_3Y6N", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of the paper is to optimize \\gamma tuning in focal loss to improve the calibration of neural networks model. The motivation makes a lot of sense. Although the novelty of the paper seems limited and idea seems adhoc, while practically the proposed approach could be helpful to the loss function and calibration study.  Some concerns to this paper are \n\n[1] The paper seems written in a rush and the paper organization and writing is weak. \n- In abstract, FLSD-53 is ambiguous to in its abbreviation when first used\n- The plots in the paper mostly have tiny legends\n\n[2] In table2, it seems AdaFocal  improved calibration but not accuracy in Cifar-10 data, while in the opposite way on Cifar-100 data, as compared to FLSD-53. Are there any interpretations about the such calibration and accuracy trade-offs?", "summary_of_the_paper": "The paper propose a new sets of algorithms to modify \\gamma in the focal loss, where a tunable \\gamma is applied at different region of model predications. Empirically study on Cifar-10 and Cifar-100 data with various of benchmark networks show better ECE as compared to the other methods. ", "summary_of_the_review": "Overall the paper proposes a new strategy to optimize calibration via tuning \\gamma in focal loss. Writing in this paper is a little concerning. The idea is a little incremental but could be potential beneficial to the community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "D-X2sRVbLav", "reviewer": "Reviewer_u2LZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strong points\n\n- The paper proposes an adaptive version of focal loss for model calibration. The hyper-parameters in focal loss can dynamically adjust for the calibration criterion.\n\n- A similar validation set is used for updating the hyper-parameters. The updating rule is simple and can be easily plugged in existing methods.\n\n- The proposed method outperforms other baselines on extensive datasets\n\n\n# Weak points\n\n- One glaring issue of the current version is the readability of figures. Fig (1),(2),(4),(5) are difficult to parse. There are even lines with duplicate colors in the second-row figures of Fig (2).b.\n\n- \"And from a calibration point of view, our strategy going forward would be to exploit this behavior to keep C_{train,true,i} (which we have control over during training) closer to A_{val,i} so that, in turn, C_{val,top,i} also stays closer to A_{val,i} to overall achieve low calibration error C_{val,top,i} − A_{val,i}\". It seems to me there are two pitfalls in the sentence: (1) since the validation set is the actual target (claimed by the author on page 2), how can people make use of the accuracies on the validation set to tune models? (2) The correspondence in average confidence $C$ can not be directly translated into the correspondence in the calibration error $|C-A|$. \n\n- It seems that the update rule in Eq. (4) is unstable. For example, if $C-A > k$ for $m$ steps, then $\\gamma \\ge e^{mk}$, which in turn makes the focal loss rather small. The authors propose to use a threshold to rein in the explosion. However, it seems to me that the explosion will commonly happen, and at most of the time $\\gamma=\\gamma_{max}$. Could the authors provide the dynamics of $\\gamma$ during the training process?\n\n# Minors\n\n- In the second paragraph from the bottom: $\\mathcal{L}_f$ should be the Focal loss L_Focal?\n\n- It seems there is an extra \"top\" in the subtitle of Fig (1).b?\n\n- Bottom paragraph on page 4: the $n$-th sample should be in $i$-th bin?\n\n- Sentence above Eq (1): A_{val,b} instead of A_{val,i}\n\n- What's the y-axis in Fig .(3)? Classification error on validation set?\n\n\n", "summary_of_the_paper": "This paper considers the problem of model calibration. Existing works calibrates the model by post-hoc approaches or objective function tailored for calibration. The authors of the paper propose an adaptive version based on Focal loss, which regularizes the overconfidence of neural networks. They observe that although focal loss improves the calibration, it leaves out the under-confident samples. To mitigate the issue, they propose adjusting the hyper-parameter $\\gamma$ in focal loss according to the model's under/over-confidence. Experiments on vision and NLP classification tasks showcase the effectiveness of the adaptive version.", "summary_of_the_review": "All in all, the idea is interesting but I find the paper in its current form to be somewhat below the standard for ICLR acceptance, especially the write-up itself. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AyCq_LSR0H", "reviewer": "Reviewer_veY9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. This paper gives an in-depth analysis of the calibration problem in deep learning. The authors investigate the calibration properties of focal loss, observe that fixed $\\gamma$ is not optimal for better calibration,  and find a correspondence to adjust $\\gamma$ in each iteration adaptively. \n2. The authors present many figures and investigation experiments for analysis and verification of effectiveness. The proposed method is effective in CIFAR10/100 and SVNH datasets. \n\nWeakness:\n1. The generalization of the proposed method is not promised. The observed problem of fixed $\\gamma$ are empirical and may vary between datasets.  On the other hand, the optimal hyperparameters for the proposed method may also vary between datasets.  Besides,  it would be better to report the large-scale datasets' results, such as ImageNet. The current experimental results are not convincing. \n2. The figures in this paper are not easy to understand. (a)There are too many figures, and hard to understand the representations of each line. Some legends even block important information. The authors may present less but important figures with concise conclusions in captions. (b) It would be better to smooth some lines in some figures. For example, figure 1 will be better if the lines are smoothed. (c)Some figures are too small to read after printing. And some figures are not readable without color printing. \n3. How will the proposed method affect the test accuracies? Will the proposed method lead to an accuracy drop as the expense for improved calibration? Figure 4 reports the accuracies but is very hard to tell the accuracies difference. ", "summary_of_the_paper": "This paper studies the calibration of deep learning, which aims to make confidence store accurately describe predictions' correctness probabilities. The authors improve focal loss and propose a calibration-aware focal loss for better calibration.  The proposed approach adaptively adjusts the coefficient of focal loss according to the momentums and current predictions' confidence. The authors conduct experiments on SVNH, CIFAR10/100 datasets to verify the approach's efficacy. ", "summary_of_the_review": "The calibration problem is interesting and practical in real-world applications.  This paper presents an in-depth analysis of the calibration problem and proposes an effective method to improve it. However, the proposed method has not been proved to generalize well in different datasets. The authors may answer this question by a theoretical analysis or experiments in large-scale datasets. Another suggestion is to improve the figures of this paper. If the authors can address my questions, I am willing to raise my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "91oyMH54nrj", "reviewer": "Reviewer_5y1y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Pros \n\n- The authors consider a wide variety of recent calibration literature.\n- The approach is intuitive and is verified by empirical results showing that the patterns between validation and training ECE seem to hold. \n- The authors motivate the need by empirically showing that a single focal loss parameter is not optimal among different bins.\n- The experiments are extensive and cover important calibration and OOD metrics.\n\n# Cons\n\n- The axes and legends in figure 1 are too small and very hard to read\n- The axes are unlabeled in figure 2a\n- Section 5: “this mostly holds true” $\\rightarrow$ It is not clear what “this” refers to. From the context it appears that “this” refers to “focal loss not leading to smaller gradients.” It appears the next sentence leads to the opposite conclusion which is confusing. \n- Figure 5 is too cluttered and extremely hard to read\n- Table 1 and the preceding paragraph: In both places, the text states that AdaFocal’s performance is averaged over 5 runs, which implies that the baselines are not. How many runs were the baselines trained for?\n- The authors highlight that AdaFocal gives the best performance among the “Pre T” results, but isn’t it also the case that AdaFocal is the only algorithm which has made use of the validation data at this point? This would explain why there is not much improvement with temperature scaling and AdaFocal.\n- The method of updating $\\gamma$ is based on summing the exponents together which may lead to large values which are mitigated by clamping to a maximum value. Have the authors considered using an exponential moving average of the exponents such that the argument to the exponential is of the form $\\alpha E_{val,t} + (1 - \\alpha) E_{val,t+1}$? I believe this would prevent the instability of the sum and not require any clamping to a maximum value.\n \n\n\n# Minor\n\n- Page 3, section 4: The sentence which begins with “Nonetheless, for completeness…” is missing a prenthesis and probably also an unnecessary comma which makes the sentence confusing to read.\n- Section 4: approaches towards $\\rightarrow$ approaches.\n- Section 5: During majority $\\rightarrow$ during the majority\n- Section 5.1: hyperparameter$\\gamma$ $\\rightarrow$ hyperparameter $\\gamma$\n- Section 5.2: this update rule do $\\rightarrow$ this updates rules does\n- Section 5.2: reign in $\\rightarrow$ rein in\n", "summary_of_the_paper": "The authors propose a new focal loss method of calibrating a model via regularization during training. They propose an adaptive method of adjusting the focal loss parameter by empirical validation performance which adjusts the parameter based on specific bin over/under confidence. ", "summary_of_the_review": "The authors do a good job in motivating their method and empirically showing why it is needed and why it should work. In mu opinion, the results are very hard to read and the paper has a problem with formatting/visibility or tables and figures. Additionally, If I have understood everything correctly, I think the formulation of equation 4 may be improved, eliminating a hyperparameter (see last bullet point in Cons section). Overall, these are the main points influencing my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work AdaFocal proposes an approach to tune Focal Loss' $\gamma$ hyperparameter to improve the model's calibration, particularly to avoid the occasional underconfidence when using focal loss. This tuning is done not as a learned constant hyperparameter across training but as one that evolves over training.

The work is both well-motivated and well-written. However, multiple reviewers share the concern (which I agree with) that the method fails on ImageNet experiments, and the method often fails to beat even temperature scaling. The experimental comparison arguing that the approach improves upon many methods pre-temperature scaling is unfair as no other method leverages the validation set. This makes for a fairly deceiving slight of hand if not read carefully. When compared to temperature scaling, which does use the validation set, the performance improvement gap is diminished altogether.

I recommend the authors use the reviewers' feedback to enhance their preprint should they aim to submit to a later venue. In particular, improve the clarity around the experimental validation.
Decision: Reject