AdaFocal: Calibration-aware Adaptive Focal Loss
Authors: Arindam Ghosh
Thomas Schaaf
Matthew R. Gormley
Venue: NeurIPS 2022 Accept (2022)
Keywords: calibration
uncertainity
out of distribution detection
focal loss

Abstract: Much recent work has been devoted to the problem of ensuring that a neural network's confidence scores match the true probability of being correct, i.e. the calibration problem. Of note, it was found that training with focal loss leads to better calibration than cross-entropy while achieving similar level of accuracy \cite{mukhoti2020}. This success stems from focal loss regularizing the entropy of the model's prediction (controlled by the parameter $\gamma$), thereby reining in the model's overconfidence. Further improvement is expected if $\gamma$ is selected independently for each training sample (Sample-Dependent Focal Loss (FLSD-53) \cite{mukhoti2020}). However, FLSD-53 is based on heuristics and does not generalize well. In this paper, we propose a calibration-aware adaptive focal loss called AdaFocal that utilizes the calibration properties of focal (and inverse-focal) loss and adaptively modifies $\gamma_t$ for different groups of samples based on $\gamma_{t-1}$ from the previous step and the knowledge of model's under/over-confidence on the validation set. We evaluate AdaFocal on various image recognition and one NLP task, covering a wide variety of network architectures, to confirm the improvement in calibration while achieving similar levels of accuracy. Additionally, we show that models trained with AdaFocal achieve a significant boost in out-of-distribution detection.

AdaFocal:
Calibration-aware Adaptive Focal Loss
Arindam Ghosh 3M Health Info. Systems Pittsburgh, PA 15217 aghosh48mmm. com
Thomas Schaaf 3M Health Info. Systems Pittsburgh, PA 15217 tschaaf @mmm. com
Matt Gormley
Carnegie Mellon University Pittsburgh, PA 15213 mgormley0cs cmu edu
Abstract
Much recent work has been devoted to the problem of ensuring that neural network's confidence scores match the true probability of being correct, i.e. the calibration problem. Of note, it was found that training with focal loss leads to better calibration than cross-entropy while achieving similar level of accuracy 19. This success stems from focal loss regularizing the entropy of the model's prediction (controlled by the parameter 7), thereby reining in the model's overconfidence. Further improvement is expected if y selected independently for each training sample Cemmlle-Dependent Focal Loss (FLSD-53) [9]). However, FLSD-53 is based on heuristics and does not generalize well. In this paper, we propose a calibration-awer aware adaptive focal loss called AdaFocal that utilizes the calibration properties of focal (and inverse-focal focall loss and adaptively modifies " for different groups of samples based on 7-11 from the previous step and the knowledge of model's under/over-confidenee on the validation set. We evaluate AdaFocal on various image recognition and one NLP task, covering wide variety of network architectures, to confirm the improvement in calibration while achieving similar levels of accuracy. Additionally, we show that models trained with AdaFocal achieve a significant boost in out-of-distribution detection.
1 Introduction
Neural networks have found tremendous success in almost every field including computer vision, natural language processing, and speech recognition. Over time, these networks have grown complex and larger in size to achieve state-of-the-ar performance and they continue evolve in that direction. Along with these advances it has also been well established that these networks suffer from poor calibration [4) i.c. the confidence scores of the predictions do reflect the real world probabilities of those predictions being true. For example, if the network assigns 0.8 confidence to a set of predictions, we should expect 80% of those predictions be correct. However, this is far from reality since modern networks tend to be grossly over-confident. This is of great concern, particularly for iission-criticss applications such autonomous driving or medical diagnosis, wherein the downstream decision making relies not only on the predictions but also on their confidences.
In recent years, there has been growing interest in developing methods for neural network calibration. These roughly fall into two categories (1) post hoc approaches that perform calibration after training (2) methods that calibrate the model during training. The first category includes methods such as temperature scaling [4, histogram binning [31. isotonic regression
(32), Bayesian binning and averaging 22 2T, Dirichlet scaling [0 mix-n-match methods (34), and spline fitting
[5). Methods
in the second category focus on designing objective functions that account for calibration during training, such as Maximum Mean Calibration Error (MMCE) [33, Label smoothing [20], and recently focal loss [19]. These aim to inherently calibrate the model during training, yet when combined with post hoc calibration further improvement is often obtained.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Contribution Our work belongs to the second category. We first show that while regular focal loss (with fixed 7) improves the overall calibration by preventing samples from being over-confident, it also leaves other samples under-confident. To address this issue, we propose modification to the focal loss, while utilizing inverse-focal loss 30 171, named AdaFocal that adjusts the Y for each training sample (or rather group of samples) separately by taking into account the model's under//ver-confidemn
about a corresponding sample (or group) in the validation set. AdaFocal also adaptively switches from focal to inverse focal loss when focal loss fails to overcome the level of under-confidence. We evaluate our method on four image classification tasks (CIFAR- 10, CIFAR- 100, Imageeea and ImageNet) and one text classification task (20 Newsgroup) using various model architectures. We find that AdaFocal substantially outperforms regular focal loss and other tf-te-offlee ffite--f-theaare art calibration aalirrationddurin- -triining techniques in the literature. Models calibrated by AdaFocal benefit more from post hoc calibration techniques to further reduce the calibration error. Finally, we study the performance of AdaFocal on an out-o--distribution detection task and find substantial improvement in performance.
2 Problem Setup and Definitions
For classification problem with training data {(x,, Mtrue.n)) where x,, is the input and Vrrue, E y = {1. K} is the ground-truth, we train a model f that outputs a probability vector p over the K classes. We further assume access validation set for hyper-parameter tuning and test set for evaluation. For example, fo(.) can be a neural network with learnable parameters 0, x is an image, and P is the output of softmax layer whose b-th element P&: is the probability score for class k. We refer to ÿ = arg maxkey pl as the network's 's prediction and the probability score P3 as the confidence. Then, model is said to be perfectly calibrated if the confidence score P matches the probability of the model classifying x correctly i.c. IP Vtrue PG = p) p, vp E [0, 1] Continuing our example, if the network assigns an average confidence score of 0.8 to a set of predictions then we should expect 80% of those to be correct.
To quantify calibration, we Calibration Error as E Pg P(O Vhrue (g)) and the Expected Calibration Error as Ep, [E] = Egg L0g - P(O Ytrue p&)1 44. Since the true calibration error cannot be computed empirically with finite sized dataset, the following approximations are generally
used in the literature. For a dataset {(x.. Mtrue. ) (1) ECEEW = -i I - -A, 44) where B, is equal-width (EW) bin that contains all examples j with Po.j in the range [ M ) C. = [.a EjEB, Pā.j is the average confidence and A; = 1 [8]]] EjEB, (0; Utrue.)) is the bin accuracy. Note that E, C, A, is the empirical approximation of the calibration error M | E, (2) ECEEM Li=1 N IC, - A.l 24 where Vi,j |B,] |BBl are equal-mass (EM) bins. Furthermore, as ECE has been shown to be biased estimate of the true calibration error (29] we additionally use ECEDEBIAS [1] and ECESWEEP 26 to corroborate the results in the paper.
1+1
3 Calibration Properties of Focal Loss
Focal loss [16] (p) = -(1 pp" log p was originally proposed to improve the accuracy of classifiers by focusing on hard examples and down weighting well classified examples. Recently, was shown that focal loss can also be used to improve calibration [19] This is because, based on the relation LFI. 2 KTgllp)- - (H) (q is the one-hot target vector), focal loss while minimising the KI. divergence objective also increases the entropy of the prediction p. This prevents the network from being overly confident on wrong predictions thereby improving calibration.
However, as we show next, focal loss with fixed y falls short of achieving the best calibration. In Figure D we plot the calibration behaviour of ResNet50 in different probability regions (bins) when trained on 11 with different focal losses. The ith bin's calibration error Eval,; Ceal, Apal, is computed on the validation set using 15 equal-mast binning. The figure plots a lower (bin-0), middle (bin-7) and higher bin (bin-14) 14) (rest of the bins and their bin boundaries are shown in Appendix B). We see that, although focal loss 7 = 4 achieves the overall lowest ECEFM, there's no single y that performs the best across all the bins. For example, in bin-O, 7 = 4, 5 achieves better calibration whereas 7 = 0,3 3 are over-confident. On the other hand, in bin-7 7 = 3 seems to be better calibrated whereas 4, 5 are under-confident and ? is over-confident.

---

Bin-0
Bin-7
Bin-14
6.02
Cuai. Aval
0.1
0.005
ECE (90
0.00
9.
100
200
300
0..4
100
200
300
100
200
300
Cooch
Spocn
Eooch
Exoch
(a) ECEEM (%)
(b) Cout.
Avut,i
Figure 1: Calibration behaviour of ResNet-50 trained on CIFAR-10 10 with cross entropy (CE), focal loss 7 3, 4, (FL-3//4/5) and FLSD-53. These are computed using 15 equal-mass binning on the validation set. (a) ECEEM. and (b) Calibration error Caal, Avat, for a lower (bin-O), middle (bin-7), and upper (bin- 14) bin. The black horizontal lines in (b) represent Eval, 0. These show that although 4 achieves the overall lowest calibration eттоr, the best performing is different for different bins.
This shows that using different ys for different bins can bring further improvement. Such attempt called the Sample-Dependent Focal Loss (FLSD-53) is presented in [9 that assigns y = the training sample's i true class posterior Pu..ue E [0, 0.2) and 3 if Prtrue E [0.2, 1]. However, from Figure FLSDD-53 also not the best calibrated method across all bins. It is strategy based on fixed heuristics of choosing higher 7 for smaller values of Pyrcuc and relatively lower for higher values of Pyrrue'
This clearly motivates the design of a strategy that can assign appropriate "S for each bin-i based on the magnitude and sign of Eval, To design such a strategy one, however, faces two challenges:
1. How do we find some correspondence between the confidence of training samples (which we can manipulate during training using the parameter 7) and the confidence of the valida- tion/test samples (which are our actual target but we do not have direct control over them)? In other words, to indirectly control the confidence of a group of validation samples, how do we know which particular group of training samples' confidence to be manipulated?
2. Given that some correspondence is established, how do we arrive at the appropriate values of y that will lead to the best calibration?
We try to answer the first question in the next section and answer to the second question leads to the main contribution of the paper: AdaFocal.
Additionally. alongside focal loss, we make use of the inverse-foctt loss 30 I7 for cases where regular focal loss fails to overcome ImageNet in section 6 and Fig. 55 where even cross entropy (y = 0) can not reach the desired level of confidence. Inverse-focal loss, plotted in Fig. Ma)), and given by
collieeeo.. See, for example, ResNet-50 trained on
LInuF (p) = (1 + p)7 logp.
(1)
serves the opposite purpose of focal loss. While focal loss reduces the over-confidence of the network, inverse-focal loss helps recover from under-confidence by providing larger gradients to the samples with higher confidences (easy samples), thereby pushing their scores even further.
4 Correspondence between Confidence of Training and Validation Samples
One way to check for any correspondences is to simply group the validation samples into M equal- mass bins (henceforth called validation-bins -bin)) and compare the confidence with the training samples that fall into the respective validation-bin boundaries. Before proceeding further, we first clarify few notations of interest.
Quantities of interest For binning the validation samples, we look at the confidence of the top predicted class ý denoted by puai,top (bin average: Cool,top). For training samples, on the other hand, instead of the confidence of the top predicted class ýj denoted by Ptrain,top (bin average: Ctrain we will focus on the confidence of the true class !/rue denoted by Peraen,trae (average: Strain,true) because during training only care about trrain,true that gets manipulated by some loss function
3

---

Validation bin-0
Validation bin-7
Validation bin-14
Avg. Confidence
1.0
CE. Gyar
1.0
1.000
0.8
3. Cure
0.975
TL3.Cat
0.8
0.6
PL 5. Chu
0.950
0..4
0.925
0.7
0.900
0
200
300
Epoch
Epoch
Epoch
Figure 2: Correspondence between confidence of training (Ctrain) and validation samples (Crat) for ResNet-50 trained on CIFAR-10 10 with focal loss y = 0 (CE), 3, 5. The binning involves 15 equal- bins where training samples are grouped using validation-bin boundaries. A lower (bin-0), middle (hin-7) and upper (bin-14) 14) bins are shown here (with the rest shown in Fig. 10 in Appendix
(Figure 8 in Appendix C compares Cerain,trus and Ctrain,top to show that as the training accuracy approaches 100%, the top predicted class and the true class become the same). For brevity, we will henceforth refer to Ctrain = Carain, truee and Ceat = Ceal,top-
In Figure 2 we compare Cirain,s validation-bin-- with Ceat, and find that there is indeed a good correspondence between the two quantities. For example in Figure 2 as increases from 0 (CE), to 3 5, the solid-line Ctrain.: gets lower, and the same behaviour is observed for the starred-line Geal, For more evidence refer to Fig. I1 12, and 13 in Appendix C where similar behaviour is observed for ResNet-50 50 and WideResNet on CIFAR- I00, and ResNet-50 on Tiny ImageNe,, respectively.
We also look at the case when training samples and validation samples are grouped independently into their respective training-bins and validation-bins. Figure in Appendix C Ctroin.i in training-bing bin-i with Coaa, in validation-bin-i. Again, we observe similar correspondence. Note here that, since the binning is independent, the boundaries of grining-binn- will not be exactly the same as that of validation-bing. but, as shown in Figure 9 they are very close to each other.
These observations, therefore, very encouraging as now we have way to indirectly control Ceat, by manipulating Ctrain,i, i.e. we can expect (even if loosely) that if we increase/decrease the confidence of a group of training samples in some lower (or middle, higher) probability region then the same will get reflected on the validation samples in similar lower (or middle, or higher) probability region. From a calibration point of view, our strategy going forward would be exploit this correspondence to push Ctrain,; (which we have control over during training) closer to Avat, (the validation set accuracy in ddiidation-bin-i so that Oval, also gets closer to A and, therefore. achieve very low calibration error C Aval, For simplifying the design of AdaFocal, we will employ the first method of common binning i.e. using validation-bins to bin the training samples.
Proposed Method
Let's denote the rath training sample's true class posterior Pan.ue by Pn and pr, falls into validation-bin b. Our goal then is to keep Pn. (or its averaged equivalent Ctrain,s) closer to Avat,b so that the same is reflected on Coai,b- For manipulating Pn. we will utilize the regularization effect of focal loss's parameter 7,. At this point, one can choose to update the of validation-bint denoted by n either based on (1) how far pr. is from Agal,b i.e. 7 = g(n, Agaat,)) or (2) how far Coat,b is from A i.e. 7 g(Cuat,b Avat.h). Such 7-update-rule should ensure that whenever the model is over-confident, i.e. Pa > Aval,& (or Goal, > Apat,)), b), 7 is increased so that the gradients get smaller to prevent Pa. from increasing further. On the other hand, when Pn. < A (or Coat,b < Avat,b). i.e. the model is under-confident, we decrease y so as to get larger gradients that in turn will increase P
It may happen that no training sample falls into a particular validation-bin. that case, Carain, is shown to drop to zero, for example in bin- 14 in Figure
"Note that for focal loss increasing ? does not always lead to smaller gradients. This mostly holds true in the region p., approximately > 0.2 (see Figure 3(a) in (19)). However, in practice and as shown by the training- bin boundaries of bin-O and bin-1 in Figure in Figure Eigure@]Aee Appendix we find majority of the training samples to lie above 0.2 during the majority of the training, and therefore. for the experiments in this paper, we simply stick to the rule of increasing ry to decrease gradients and stop Pn from increasing and vice versa.

---

25
FL3
20
Errol (56)
ECE2 178)
LO00
CF-100
Re
R.50-53
FL50 53
0.7
CF-100
CF-10
C7-100
CF-10
0.
84
0.
Fooch
Epoch
100
100
Loss functions
(6) Error (%)
(c) K0B (8)
(d) Ctranss Coats Aust
Figure 3: (a) plots different loss functions: Cross Entropy (CE), Focal loss (FL-7), Inverse-Focal loss (InvFL-7). CalFocal (CF-X) with Aral, = 0.8. Subfigures (b) and (c) plot error and ECEgM- respectively, for ResNet-50 trained
CIFAR-10 with CalFocal. (d) compares Cirain (solid lines), Ceat (starred lines) and Aeal (dashed lines) in validation bin-O to show that when CalFocal brings Cirain closer Apat, Cvol also approaches Aval-
To test this strategy, we first design calibration- aware y-update method (called CalFocal), which with some additional modifications will lead to the final AdaFocal algorithm.
5.1 Calibration-aware Focal Loss (CalFocal)
Case 1: 7 - g(pn A Treating A the point that want Pn. not deviate from, make the focal loss parameter a function of pn - Avat,b to get
(CotFoou((P.) = (1 11-pri pn) log Pr: where Tn = exp(A(pn - Aval.))),
(2)
and b is the validation-bin which Pa falls. The hyper-parameter À is the scaling factor which combined with the exponential function helps to quickly ramp up/down 7. The exponential function adheres to the 7-update rule mentioned earlier. Figure33pp plots Focll Vs. Py for A = 0.8. We see that based on the strength of X, the loss drastically drops near pn 0.8 and thereafter remains close to zero. This shows that LoalFoeol aims is to first push towards 0.8 and then slow its growth towards overconfidence. Next, in Figure C)., reduce the calibration eгror (ECEgM) but it is still far from FLSD-53' 53's performance. Also note in Figure B that too high (=100) affects the accuracy of model. Most interesting Fig. 3)d) which compares Crroén,i with Cvat, (and also Avat,s) for bin-O, where we find evidence that the strategy of bringing Pa or Orrain, (solid lines) closer to Apal.; (dashed lines) results in Crai, (starred lines) getting closer to Aval,i as well, thus reducing the calibration error Eval, = Coal,i Aval,i.
find that CalFocal with À 10, 100 is able to
Case 2: y (CCeat A Note that Eq. 2 assigns a different Yn for each training sample pn- To reduce computation and avoid keeping track of "/n for each training sample, to each validation bindattion--in-b by making it function of Crat,b - Ayal, (instead of pa - Aval,p). Then all Pa that fall into the validation-bin-A are assigned 76 and the loss function is modified
assign "
Foct (p.) -(1 p.)) 7h log Pr: where % exp(AACCaat.bb Avat,b))
(3)
b is the validation- bin in which p. falls. The performance of this strategy, shown in Appendix P is
very similar (or slightly better than) Eq. 2 Further, makes more sense to update y based on how far Geat is from A instead of how far p.. is from Aval,b because, shown in Figure 3 bin-O,
one may find Cval.6 (starred lines) quite closer to Aval,& (dashed lines) even when Pr. or Ctrain (solid lines) is still far from Aval, b- At this point where Cval,b Avai,s we should stop updating 7 further, even P.. Ayat, * O, as we have reached our goal of making Emat, = Cpat, A = 0. Therefore, for AdaFocal will use Eq. 3 as the base for AdaFocal loss function.
(1) Let's say at some step of training, a high over some epochs reduces the error Cral, Agal,b. Then, it is desirable continue training with the same high Yo. However, note CalFocal's update rule in Eq. 3 whiichh will reduce + 1 as the Crat.s - Avat,s + 0. (2) Let's say, at some point Cuaa,y - Avat, & is quite high. This will set ?t to some high value depending on the hyper-parameter X. Assuming this 7h is still not high enough to bring down the confidence, we would want a way to further increase 7b. However, Calaocal incapable of doing so as will continue to hold at 7 exp(A(Coul, Avat,b))
Limitations of CalFocal
5

---

5.2 Calibration-aware Adaptive Focal Loss (AdaFocal)
We propose to address the above limitations by making 74 depend on 1t-1 from previous time step
Nt. = 1,6 exp exp(AACral.b - Aval,b))
(4)
This update rule address the limitations of CalFocal in the following way: let's say at some point we observe over-confidence i.e. Feoat = Grat Avat, > O. Then, in the next step 7 will be increased. In the subsequent steps, it will continue to increase unless the calibration error F starts decreasing (this additional increase in y was not possible with CalFocal). At this point, if we find Evat, start decreasing, that would reduce the increase in " over the next epochs and 7/6 will ultimately settle down to a value when E = (CalFocal at Lval. = 0 will cause " to go down to 1). Next, if this current value of "/ starts causing under-confidence i.e. Cool.6 Apal,b 0, then the update rule will kick in to reduce y thus allowing Ceal. to be increased back to Aval,b- This oscillating behaviour of AdaFocal around the desired point of v = Aval,b is its main strength in reducing the calibration error in every bin.
Next, to deal with cases where even cross entropy suffers from under-confidence, we switch to inverse- focal loss which can further increase the confidence of the predictions. For the switch between focal and inverse-focal loss, we simply set threshold Sth. below which if gamma falls, we switch to the other loss function. Note here that, for notational purpose, we will denote the inverse-focal loss by a negative 7 i.e. Nt > 0 means focal loss with parameter "t, whereas "t, < 0 implies inverse-focal loss with parameter [tt.l The complete gamma-update rule and the switching criteria is given in Algorithm| If not stated explicitly, we use Sn, 0.2 for all AdaFocal experiments. For reference, we also plot results with Sn = 0.5 for ImageNet, ResNet-50 in Fig. (d) and Fig. 5jb).
Finally, note the unbounded exponential update in Eq. which is an undesirable property. This may easily cause "t to explode as when expanded 7t - exp(Feat, :) = "to (ep(FFatt, + Erad,1 + ... + Feat,t-1 + E) ), and if Exat,t > 0 for quite few number of epochs, "/e will become so large that even if Eval,t < in the subsequent epochs, it may not reduce to desired level. We remedy this by constraining //t to an upper bound "mmxx when > 0 (focal loss) and lower bound of Tmin when y < (inverse-focal loss). Therefore, the final AdaFocal loss is given by
-(1 pv. pro log Pr: if 7t,b 0 AdoaFocal (Pr, t) [-11+ p..) hu log
(5)
if 75,8 < 0,
and the complete algorithm along with the gamma-update rules is given in Algorithm 1 is presented in the next section.
A discussion
on the selection of
Experiments
We evaluate the performance of our proposed method on image and text classification tasks. For
image classification, we use CIFAR-10, CIF g -may-Im (2), and ImageNet [27 to analyze the calibration of ResNet50, ResNet- 100 6 Ride-Res 10 33), and DenseNet-121
8 models. For text classification, we use the 20 Newsgroup dataset [1] and train a global-pooling CNN 15 and fine-tune pre-trained BERT model [3]. More details about the datasets, models and experimental configurations are given in Appendix D In the main paper, we report results using only ECEEM, whereas other ECE metrics are reported Appendix.
Baseline Among culibrationddurigg-rraann training methods we use MMCE 13, Brier loss 0. Label smoothing (L-S-0.05) (20) and sample-dependent focal loss FLSD-53 as baselines. For post hoc calibration, we report the effect of temperature scaling, ensemble temperature scaling (ETS) [34] and spline fitting (5) on top of these methods. For temperature scaling, select the optimal temperature E (0, 10] (step size 0.1) that gives the lowest E'CEEM on the validation set.
Results In Figure we compare AdaFocal against cross entropy (CE) and FLSD-53, for ResNet- 50 trained on various small to large-scale image datasets. Among various focal losses, we chose FLSD-53 as our baseline because it was shown to be consistently better than MMCE, Brier Loss and Label smoothing [19] across many matasets-model pairs. The figure plots the test set error and ECEEM. In 5 for ResNet-50 on CIFAR-10 10 and ImageNet, we plot (1) the calibration statistics Fuat = Cval - Agai used by AdaFocal during training and (2) the dynamics of y for few bins in
lower, middle, and higher probability regions.

---

Neal. Algorithm 1: AdaFocal
Input:
Dtrain {xx: Utrue |==1 Deal {xn: Strue I=
2 Bin initialization: for to M do
o, (==l M. 7 = // Initialize validation-bins equal-width with gamma
set 1;
Training: for / = 0 to T do
Loss; 0;
for n to Ntrain do
Pn = fo, (Xa) // Denoting Puerue.. by Pni
b = bii__nddex(pa {Bi,i) // Bin in which pa lies;
if Yr.h 2 0 then
10
Loss; += -1 Pr. log
Focal loss;
else if dt,b < 0 then
12
Loss; += - (1 + Pr) hes log Pr
// Inverse-focal loss;
13
0.+1- gradient_ cectiinnt_upatee((( (0,, Loss:);
-update step: for
to M do
15
Re-compute B,+1, Ceal.++1, and Aval,t+1, using the updated model for4 and Deali
16
if Tr, 2 0 then
7+1, = min| 7max: Te * MCCoat,ess Aoatatttssssssts..a // Focal loss y-update;
17
18
h+41sl | < Sen then
1,,,, SSh Switch to inverse-focal loss;
20
else if Yo. < then
701, = max { Tmin, Yes * MC // Inverse-focal y-update; if h141al < Sen then
21
22
741, Sch // Switch focal loss;
First, we observe that for CIFAR-IO, CIFAR-100 and Tiny-ImageNor FLDS-53 is much better calibrated than CE. This because, as shown in Fig. (Ta for ResNet-50, CIFAR-I0. 10, CE is over- confident compared to FLSD-53 in almost every bin. For ImageNet, however, the behaviour is reversed: FLSD-53 poorly calibrated than CE. This, as shown Figure is due to the use of high values of (= 5, 3) by FLSD-53 which makes the model largely under-confident in each bin, leading to a overall high calibration error. AdaFocal, on other hand, maintains a well calibrated model throughout the training for all cases.
Next, for ResNet-50 trained on CIFAR-10, 10, find 7t to be closer for higher bins and closer to 20 for lower bins. These S result in better calibration than = 5, 3 of FLSD-53. For ImageNet, on the other hand, find, except bin-14, 14, AdaFocal switches inverse-focal loss some point in the training. This makes sense because for ImageNet even cross entropy (?x = 0) suffers from under-confidence, therefore, AdaFocal, starting from Yt first approaches cross entropy (y = 0) to ultimately switch to inverse-foce loss (% < 0) at Ss, = 0.2. Since, in retrospect, we already know that switching to inverse focal loss is beneficial for ImageNet, switching early at St.. 0.5 helps to reach the same level of calibration early. Overall, these experiments confirm that during training AdaFocal being aware the network's s current under/over-confidence able to guide the 7:8 to values that maintain well calibrated model at all times. Also note that for unseen dataset-model pair there's 's no way to know apriori which y will perform better but AdaFocal will automatically find these appropriate values thus avoiding expensive and extensive
search.
Comparison of ECEEM and test set error on for rest of the experiments shown in Table I and 2 reeppecttely From Table I we observe that prior to temperature scaling, AdaFocal outperforms the other methods in 14 out of 15 cases. Post-temperature scaling. AdaFocal achieves the lowest calibration eтror in 12 out of the 15 cases. Further, observe that the optimal temperatures are mostly close to indicating that AdaFocal produces inherently calibrated models from training itself. Effects of other post hoc calibration methods (ETS and spline fitting) are shown in Appendix F Again, we observe that pre-calibrated models from AdaFocal benefit from post processing to further lower the overall calibration error. Besides ECEEM. the consistency of the results across other calibration
7

---

15
-5
RC (86)
Enar (16)
10
ECEx
FLSD-53
FLSD-53 Adaaooal
Adefocal
200
100
200
300
100
200
300
100
200 Epcch
300
Epoch
Epoch
Epoch
CIFAR-10
(b) CIFAR- 100
CE
15
FSD
Adeloccal
ECE (70
Errur (%5)
<0
AdaFocal Se
Adatocall Sy U.
02
BC5 (%)
Adal 5y 0.5
FL5D-53
Adwltoaal Su,
8
5
20
10
60
100
20
60
00
103
Epoch
Cpochs
[pncn
Epech
(c) Tiny-Imagett ImageNet
(d) ImageNet
Figure 4: ResNet-50 trained with cross entropy (CE). 53 and AdaFocal. In each subfigure, left: Error (%), right: ECEFM (%) on the test set are plotted for mean and standard deviation over 5 runs. We observe that throughout the training AdaFocal maintains a low calibration error while achieving similar accuracy.
Datase
Model
Fre Temperature scaling
Fost Temperature acaling
CE
Brier
MMCE
L.S
FLSSD-53
AddFosii
CF
Brice
MMCE
E.
FLSDD53
AdaFocal
ResNee50
4.24
1,78
4.32
3.85
1.53
0.86
2.11(2.52)
124(1111)
2.222.51
2.990.991)
1/421 (8)
0.36:196)
CIPAR-10
WideResNNe
RosNei10
3.42
4.39
2,63
1.72
5.16
3.31
4.44
4.26
1.90
1.82
0.64
0.71
2.2712.74)
1.8712.16)
1.221100)
1
2.53(2.83)
1.62222)
4,4411.00 00)
2.440..11
1.570.94)
122111200)
0.46:1966
6.7311002
21
4.26
2.09
5.05
4.00
1.40
0.62
2.2112.33)
20001000
2.252.552
3.3100.94)
1.411.000)
0.5%1.023
ReaNa50
17.17
6:37
15.25
7.85
5.54
1.36
3.7112.16 16)
386:113
2331180
4.161.13)
20711117
1.36 1.000
CIBAR 100
WaleResNes
Ro
14.53
19.44
4.27
7.70
13.12
19.11
11:15
5.10
2:25
7.00
1.95
1.40
3.23/2.12)
6.1112.285
2.851108)
4.55
4.8322332
(2201.91)
8.58r1 (9)
5.1011.000 000
3.8511.200
2.2511.000
1.95:1000
1.40
D2msNNe121
19.82
5.34
19.16
12.81
2.58
L.73
3.6212.2))
2.58:1009
5.1122113)
8,9511.19)
1.8011.10)
1.731-000
Tay ImageNet
1o
ReaNe130
8.11
7.81
3.74
1.42
7.40
8.49
9.12
9.35
2.36
1.88
1.61
1,85
1,811.20
2.83/991)
23K0933)
4.251..95 361
1.95(120)
4.860 78)
4.51
2 (8)
5811 (00
1.2%
1.851.900
ReaNa33
2.23
3331
0:30
10.05
16.77
137
1.500888)
3.5900922)
2.221.344
4.336033)
26200.74
13 1M0
ImugeNNe
RoNNel10
1.28
3.98
1.83
4.02
18.66
1.17
2.8708500 1.2811300
1.8341.00)
2.7660.90)
2.5100.70
1.17(1000)
D2
1.82
2.94
1.23
5.30
19.19
1.50
1.521.000
2.21/0.00
1.2201.000
1.420.900)
2.2600770
1.50010000
MNewggroup
BERT
CSN
1857
8.47
1332
5.91
1523
8.30
6.01
4.36
8.30
8.83
1.62
3.96
44661.44)
4.801.78)
31 922335
4.4011244
4 1.001-45
6.502.211
2.891 12)
3.691.14)
3,91/0.00)
2.1311336
2.464 2.46x11000 1000
3.7311944
Table 1: Test ECEEM (%) averaged over 5 runs. Bold marks the lowest in pre and post temperature
scaling groups separately. Optimal temperature, given in brackets, is cross-validated on ECEEM-
metrics is shown through ECEDEBIAS, ECEsWEEr (equal-width and equal-mass) in Appendix F Statistical significance of the results is confirmed through ECCEg error bars in Appendix where mean and standard deviations are plotted over 5 runs.
(OOD) detection Following 19 we report the performance of AdaFocal CIFAR-10 as the in-
on OOD detection task. We train ResNet- and Wide-ResNee26- distribution data and test on SVHN (23) and CIFAR- 10-- [7] (with level 5 Gaussian noise corruption) as OOD data. Using entropy of the softmax as the measure of uncertainty, the corresponding ROC plots are shown in Figure 6 and AUROC scores are reported in Table ( in Appendix K We see that models trained with Adaroca outperform focal loss 7 3 (FL-3) and FLSD-53. For the exact AUROC scores, please refer to Appendix K These results further highlight the benefits of inherently calibrated model produced by AdaFocal as post-hoc calibration techniques such as temperature scaling, as shown in the figure, is ineffective under distributional shift (2
prree--prrmeters The perameearsmeters introduced by AdaFocal are X, Tmin max- and Sih- However, these do not require extensive hyper-parameter search and are much easier select compared to (which, otherwise, needs to be searched for every bin at every time step). Based on our experiments,
À is redundant and one may choose to ignore it as for all our experiments À = 1 worked very well. However, for an unknown dataset-model pair, if increasing/decrrasing the rate of change of ? improves calibration (or helps to reach the desired level faster), then one can use À to achieve
8

---

Bin 14
Bin 10
Bin 14
Bin 10
0.01
0.01
0.10
0.10
Cusl - Aual
Aval
CE
CF
0.00
FLSD-53
Adarocal
Adefocal
-0.02
200
300
-0.02
200
300
-0.10
20
60
-0.10
20
40
50
1.4
1.0
1./ aca 5.-05
1.2
0.5
2212
1.0
Yu AdaFocal
AdaFocal
1.1
1.0
Bin
Bin 0
Bin 5
Bin
0.10
9.10
Cual - Ava!
Aval
0.05
0.05 /f
0.00
0.00
40
0.0
0.0
FLSD-63
0.05
FLSD-52 Asshocal
Ju UAA
Adshocal
-0.10
100
200
300
-0.10
100
200
300
-0.20
20
80
80
-0.2
20
40
60
1.8
Ye. Adallocal
Adal ocal
S.-02
22 1.4
1.2
100
200
300
100
200
300
20
50
Eooch
Epoch
Epocn
Epoch
(a) CIFAR-10, ResNet-50
(b) ImageNet, ResNet-50
Figure Dynamics of ys in different validation-bin when ResNet-50 is trained on CIFAR-10 and ImageNet with AdaFocal. For each bin, top: Coal A from the validation set, bottom:
Tt vs. epochs. Black dotted line in top plot represent zero calibration error. We observe that for each bin AdaFocal is able to find the gammas that result in low calibration error. For CIFAR-10,
" > whereas for ImageNet, starting from "t- (focal loss), AdaFocal ultimately switches to inverse-focal loss (%2 < 0) S. 0.2 (or 0.5) for some of the bins.
Dataset
Model
CE
Brier
MMCE
LS
FLSD-53
AdaFocal
ResNet50
4.95
5.00
4.99
5.29
4.98
5.30
ResNet1 10
4.89
5.48
5.40
5.52
5.42
5.27
CIFAR-10
WideResNet
3.86
4.08
3.91
4.20
4.01
4.50
DenseNet121
5.00
5.11
5.41
5.09
5.46
5.20
ResNet50
23.30
23.39
23.20
23.43
23.22
22.60
ResNet1
22.73
25.10
23.07
23.43
22.51
22.79
CIFAR-100
WideResNNtt
20.70
20.59
20.73
21.19
20.11
20.07
DenseNet121
24.52
23.75
24.0
24.05
22.67
22.22
ResNet50
42.90
46.27
45.96
44.42
45.12
45.49
Tiny-ImageNet
ResNetl
42.53
45.47
42.22
44.13
44.88
44.55
ResNet50
27.08
28.80
27.12
28.43
28.53
27.07
ImageNet
ResNetl 10
23.77
24.07
23.72
23.84
25.17
23.66
DenseNet121
27.84
28.02
27.87
27.79
29.12
27.74
CNN
26.68
27.06
27.23
26.03
27.98
28.53
20Newsgroup
BERT
16.05
16.52
16.16
16.18
17.57
17.22
Table 2: Test set error (%). Lowest error is marked in bold.
Yinin do not require any special fine-tuning as their sole purpose is to stop y from exploding in either directions. This is similar the common practice of gradient clipping for stable training. For all our experiments, we use "max = 20, but any reasonable value around that range should also work well in practice. For comparison of results with "max 20, 7max 50 and " oc, please refer to Appendix(N) N
Tmin 2 is selected based on the observation that values beyond 2 led to unstable training. However, if, for new untested dataset-mode: pair, Tmin 2 turns out to be unsuitable, it should

---

14
14
TL3, Pre
FL-3,
FL-3 Pre
FL3. Pre
[L33, Post
TL:3, Post
FL.3
TL33 Post
FLSD 53, Pro
FL5D 53, Pro
RL50.53, Pro
FLSD 53, Pre
0.2
1150-55, Plal
02
1L5U--5, Poal
04
1L50155, Pool
1150-55, Pout
AdaFocal
Adaifocal
Adeicccal
Adeioccal
100
1.1
FPR
14
00
162
111
100
14
24
(a) SVHN: ResNet- 110, WideResNet
(b) CIFAR- 10-C: ResNet-110, WideResNet
Figure 6: ROC for ResNet-110 10 and Wide Relle-ReNNet22-111 10 trained in-distribution CIFAR-I0 and tested on out-of-distribution (a) SVHN and (b) CIFAR-IO-C. Pre/Post T refers to pre and post temperature scaling.
still be fairly easy to select new threshold by simply looking at the "dynamics of y" plots (similar to Fig. 5) at the time step the training becomes unstable.
We use Sta 0.2 for all our experiments. This also does not require extensive tuning and can be easily selected based on the evolution of in various bins for a trial run of AdaFocal ( without switching to inverse focal loss). For example, for ImageNet (Fig. ): where some of the bins are always under-confident AdaFocal decreases y (starting at 1) towards negative values. In this case, it makes more sense have a higher S.,, (= 0.5) so that AdaFocal can switch early to inverse-focal loss and overcome the under-confidence (see ImageNet results in Fig and
Number of bins. We experimented with AdaFocal using 5, 10, 15, 20, 30, and 50 equal-mass bins during training to draw calibration statistics form the validation set. As reported in Appendix H the best results are for number of bins in the range of 10 to 20. Performance degrades when the number of bins are too small (< 10) too large (> 20). Therefore, we use 15 bins for all AdaFocal trainings. Note that for computing ECE metrics as well, we use 15 bins so as to be consistent with previous works in literature 19
Best choice of AdaFocal post-hoc calibration From the results in Table and 4 for tempera- ture scaling, ETS and Spline fitting, respectively. there not a clear choice of post-hoc calibration method that gives the best results across all dataset-mont modeel pairs when combined with AdaFocal. However, we observe that in almost all cases it the lowest ECE when combined with one of these post-hoc calibration techniques. Therefore, for an unknown mataset-model pair, the choice of the best post-hoe calibration method (to be used on top of AdaFocal) might require more investigation and is different study in itself. Overall, the evidence in our paper show that pre-calibration from training leads to even better calibrated models post hoc, and AdaFocal is much better in producing such pre-calibrated models.
pre-calibrated model by AdaFocal that gives
7 Conclusion
In this work, we first revisit the calibration properties of regular focal loss to highlight the downside of using fixed 7 for all samples. Particularly, by studying the calibration behaviour of different samples in different probability region, we find that there's no single that achieves the best calibration over the entire region. We use this observation to motivate the selection of y independently for each sample (or group of samples) based on the knowledge of models's 's under/over-confidenec from the validation set. We propose calibration-awwe aware adaptive strategy called AdaFocal that accounts for such information and updates the " at every step based on "h-1 from the previous step and the magnitude of the model's under/over-confidene across different dataset-model pairs producing inherently calibrated models that benefit further from post-hoc calibration techniques. Additionally, we find that models trained with AdaFocal are much better at out-of-disse distributiio detection.
We find AdaFocal to perform consistently better
10

---

References
[1] Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 1950.
[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, 2009.
[3] Jacob Devlin, Ming-Wi Beng-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[4] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2017.
[5] Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchis- escu, and Richard Hartley. Calibration of neural networks using splines. International Conference on Learning Representations, 2021.
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[7] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruttions and perturbations. In International Conference on Learning Representations, 2019.
[8] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[9] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[10] Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, 2019.
[1]] Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, volume 32, 2019.
[12] Aviral Kumar. 20 newsgroups mmce. https: http//gtttbb.. 2018.
com www.tau....maavriiaeaeee wwtp./vwwwaaaallaaa220777777777 /MCCE
[13] Aviral Kumar. Sunita Sarawagi. and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2018
[14] Ken Lang. Newsweeder: Learning filter netnews. In in Proceedings of the 12th International Machine Learning Conference (ML95, 1995.
[15] M. Lin, Q. Chen, and S. Yan. Network network. CORR, 1312.4400, 2014.
[16] Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
[17] Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional adver- sarial domain adaptation. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[18]
Jishnu Mukhoti. Focal calibration.
calibration 2020.
[19] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip HS Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, 2020.
[20] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, 2019.
11

---

[21] Mahdi Pakdaman Nacini and Gregory F Cooper. Binary classifier calibration using an ensemble of near isotonic regression models. In Data Mining (ICDM), 2016 IEEE 16th International Conference on, 2016.
[22] Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Niing AAAI Conference on Artificial Intelligence, AAAI 15, 2015.
[23] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[24] Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2015.
[25] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2014.
[26] Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration emrrr estimation, 2021.
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV)), 115():1113 2015.
[28] Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncer- tainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
[29] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In Proceedings of the Twenty- Second International Conference on Artificial Intelligence and Statistics, volume 89, pages 3459-3467. PMLR, 16-18 Apr 2019.
[30] Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, volume 34, pages 11809-11820 Curran Associates, Inc., 2021.
[31] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. ICML, 2001.
[32] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In KDD, 2002.
[33] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
[34] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning, pages 11 11117-11128. PMLR, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] in Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] in Ap-
pendix A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
12

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? (es]] As part of the supplementary material and details are mentioned Appendix D
(b) Did you specify all the training details (e.g.. data splits. hyperparameters, how they chosen)? [Yes] Appendix D
(c) Did you report error bars (e-g.. with respect the random seed after running experi- ments multiple times)? [Yes] see Appendix G
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] in Appendix [
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes] in Appendix D
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "TZmt5njKTRy", "reviewer": "Reviewer_gWGg", "summary": "The paper proposes AdaFocal, an adaptive focal loss that modifies the focal loss hyperparameter $\\gamma$ for different sample bins based on historic values of $\\gamma$ and the model's per-bin calibration properties. The authors evaluate AdaFocal on several image classification benchmarks (CIFAR-10, CIFAR-100, TinyImageNet, ImageNet) and one text classification dataset (20 Newsgroup).\n\n**Implementation details**\nThe $\\gamma$ update rule adjusts $\\gamma_{t+1}$ based on the value of $\\gamma_{t}$ and the difference between the model's accuracy and confidence in a particular validation bin. Training examples are assigned to particular validation bin based on validation bin boundaries that are re-computed once per epoch. The adjustment for $\\gamma_{t+1}$ is then computed based on the validation bin miscalibration that the training example is assigned to.", "questions": "- The validation bin boundaries are only updated once per training epoch. What happens if they are updated more frequently (i.e. every minibatch or at least a few times per training epoch)?\n", "limitations": "No, the authors should address limitations and potential negative social impact of their work.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The update rule for gamma and the algorithm for AdaFocal is fairly straightforward. The overall simplicity will ease implementation and increase adoption of the method.\n- The authors give strong empirical evidence that the confidence of the neural network on the training data roughly matches the confidence of the validation data for the image classification datasets, which helps motivate the AdaFocal method.\n- The empirical results for AdaFocal are strong across all experimental setups for pre-temperature scaling and strong for CIFAR-10/CIFAR-100 for post-temperature scaling.\n\n\nWeaknesses:\n- The method still involves some heuristics which may not generalize to other datasets/tasks and the paper uses empirical evidence collected only on a limited selection of datasets/tasks to justify the method.\n  - For example, one such heuristic is thresholding $\\gamma_{t}$ based on a pre-set hyperparameter $S_{th}$ to switch between the inverse-focal and focal loss. Another heuristic is the number of bins. Both of these hyperparameters may vary in optimal value based on the dataset, model, and task.\n- Only one model architecture is tested for TinyImageNet, ImageNet, and Newsgroup 20. \n- Post temperature scaling, other methods outperform AdaFocal on both ImageNet and Newsgroup 20 tasks.\n- Error bars should be included in Figure 4 if the plots are showing an average over 5 runs.\n- The plots comparing confidence of training and validation samples should be included for the Newsgroup 20 classification task.\n- Missing baseline comparison to NeurIPS '21 calibration-during-training method \"Soft Calibration Objectives for Neural Networks\" (https://arxiv.org/abs/2108.00106)\n\nMy main reason for leaning towards a borderline reject rather than accept is that the method is justified purely based on empirical evaluation but the empirical evaluation has several weaknesses (mentioned above).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OqP1OfpkrQK", "reviewer": "Reviewer_9C4n", "summary": "This paper proposes a focal-loss-based method called AdaFocal for uncertainty calibration problem. When training models using AdaFocal, the hyper-parameter $\\gamma$ used in focal loss can be dynamically varied during epochs and different samples.\n\nThe experimental results show the effectiveness of AdaFocal for calibration in both the cases of with and without post-hoc calibration. The experiments also shown AdaFocal can easily align mean confidence with accuracy of each bin in validation set.\n", "questions": "See weaknesses.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed AdaFocal method exploits an adaptive strategy to determine the important hyperparameter $\\gamma$ used in focal loss. AdaFocal can be considered as a generalized version of naive Focal loss and Focal-53.\n\n2. The illustrative experiments before introducing the proposed method give clear motivation.\n\n3. The datasets used in experiment part are from different domains, while have different difficulty degree. The results show the effectiveness of the proposed method.\n\nWeaknesses:\n\n1. The proposed method is used to adaptively determine the hyperparameter $\\gamma$ used in Focal loss. However, other hyperparameters are induced in the proposed method like $\\lambda$ and $S_{th}$. Would these hyperparameters significantly impact influence the final performance? Given this more complicated method, one naturally expects that the choosing of new hyperparameters like $\\lambda$ and $S_{th}$ is much easier than that of $\\gamma$.\n\n2. Figure 2 shows the good correspondence between the average confidence of training set and validation set. In this case, the training samples are grouped using **validation-bin boundaries**. I think a fairer setting is that the training samples and validation samples are grouped independently into their respective training-bins and validation-bins. But in this case (as is shown in Figure 9) the correspondence between them is not good.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GIeneJt5_gW", "reviewer": "Reviewer_J28C", "summary": "The authors present an adaptation of Focal Loss that adapts the focal hyperparameter over the course of training to the data. This produces on average better calibrated models than traditional cross entropy or focal loss, especially when combined with post-hoc recalibration.\n\n", "questions": "- l. 76 what is $\\mathbf{H}$ again?\n- Can you reason why for ImageNet Post Temperature Scaling CE produces the best calibrated model?", "limitations": "not applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Incidentally, it also removes the focal hyperparameter from consideration.\n- The authors introduce problem and related concepts well.\n- The results section is comprehensive within their chosen focus and concise.\n- The work overall is novel and relevant as model calibration is important for reliability.\n\n**Weaknesses**\n\n- The authors mostly focus on image classification. Some of their observations may not hold for other scenarios. e.g. their hyperparameter $\\lambda$ might become relevant.\n- Section 3 and following would benefit from some editing.\n- There are a few naming inconsistencies e.g. calibration error and calibration gap. Consistency here makes it easier to read.\n- The explanation of how the validation binning is applied to the training samples could be expanded.\n- There are some typos (FLDS, parametera) and I think the qualities you describe as \"innate\" are more inherent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0i4HmZdNFW", "reviewer": "Reviewer_2r5v", "summary": "This paper proposes AdaFocal, a scheme to dynamically pick the $\\gamma$ weight of the focal loss for the different groups of samples when training so that the expected calibration error is minimized w.r.t. validation set, during training a classifier. The weight $\\gamma$ were adjusted so that the focal loss can suggest a classifier to be overconfident or underconfident in a certain region to calibrate the confidence to be closed to the mean accuracy. Experiments showed that AdaFocal outperformed many existing methods for calibrating the classifier, including the recent focal loss-based calibration method (FLSD-53) by Mukhoti et al. ", "questions": "1. It is unclear to me what the weakness of AdaFocal is. Is it possible that training with AdaFocal may require a longer training time than using CE, etc.?\n2. There is also a recent theoretical work [A], where they conducted a rigorous theoretical analysis of the behavior of $\\gamma$ of focal loss and its relationship to calibration, which suggested that focal loss is not estimating class posterior probability. Also, it has been shown in their work that in a non-deep setting (Table 2 of [A] in the appendix: linear model or simple multilayered perception), using focal loss is worse than vanilla cross-entropy. I was wondering how is the performance of AdaFocal in such a simple setting. \n3. SVHN is also a dataset that standard CE seems to be better than Focal loss in terms of calibration even in the deep learning setting (according to [A]). In AdaFocal paper and Mukhoti et al. papers, SVHN is not used. In AdaFocal paper, it is used in OOD detection but the ECE score is not presented in the main body of the paper. I was wondering if AdaFocal can mitigate this drawback of the focal loss (I feel AdaFocal may work well in SVHN but Mukhoti's method might not), because this loss use validation set to enforce the confidence is always calibrated w.r.t. calibration error. \n4. Regarding the weakness of AdaFocal from the theoretical perspective highlighted in the \"Weakness\" section, have the authors considered the theoretical perspective of AdaFocal, perhaps something similar to the analysis in [Mukhoti et al.] or [A]? \n5. Did the authors apply early stopping to the learner, since each method may have a different time to achieve best performance? For example, in Tiny Imagenet according to Figure 4, CE achieves the best performance around epoch 40 (where other methods also performed pretty well there). On the other hand, CE performance degraded as the epoch increases and temperature scaling may not be able to a reliable confidence score.\n\nMinor\n1: Table 8 in Appendix is an empty table. Please remove it or the authors may have forgotten to put the table information.\n\n\n[A] Charoenphakdee, N., Vongkulbhisal, J., Chairatanakul, N., & Sugiyama, M. (2021). On focal loss for class-posterior probability estimation: A theoretical perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5202-5211).", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Significant and originality\n- To my knowledge, the proposed method is novel, and its performance of the proposed method is impressive. \n\nStrengths\n- Proposed method is relatively simple to implement although there are several hyperparameters. It seems the standard choice of hyperparameters is applicable to many datasets at the same time (at least for image datasets in the experiments).\n- Experiments on OOD detection have also been conducted, where AdaFocal is superior to other methods.\n\nWeaknesses\n- No theoretical aspect of the loss is discussed. It is uncleared what is happening to the minimizer of this loss function. In my understanding, the method is quite heuristic, the choice of weighting function in Algorithm is a little bit arbitrary (using exp(\\lambda). Nevertheless, I think the intuition of the proposed method makes a lot of sense and quite interesting to try. \n- AdaFocal introduces several hyperparameters to the model training process, e.g., number of bins to use, max/min of $\\gamma$\n- AdaFocal also introduces more training time since it has to evaluate the validation performance on each bin in every epoch (maybe not a big drawback in normal case) \n- The accuracy of using AdaFocal is not guaranteed to be the same compared with standard training with CE + postprocessing methods like temperature scaling. Although it is unclear when AdaFocal can have better performance than CE (and vice versa: see CIFAR-10 ACC and CIFAR-100 Acc). For anyone who simply wants to use the same loss function (because the model has been using CE for a long time and many promising hyperparameters for CE may not be the one for AdaFocal) , using AdaFocal may not be the way to go.\n- (minor) Reliability diagram result is not provided. I think looking at the reliability diagram can also be informative (at least in the appendix for future reference would be nice).\n\nClarity:\n- The paper is well-written overall and not difficult to follow. However, I found that figure is a bit difficult to read and takes some time to understand. But the information provided in the paper is enough to understand the figure.\n\nOverall:\nI think the paper has sufficient evidence to suggest that using different $\\gamma$ for different bins can be effective when using the Focal-loss style technique for improving the calibration of the classifier. The idea of using an over-confident version of Focal loss for calibration problems is not that straightforward in my opinion, and could potentially be used for a different problem in the future. Thus, I feel the paper has merits that outweigh its drawbacks, which are a lack of theoretical evidence and the increasing number of hyperparameters for training a model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept