A Note On The Stability Of The Focal Loss
Authors: Martijn P. van Leeuwen
Koen V. Haak
Gorkem Saygili
Eric O. Postma
L.L. Sharon Ong
Venue: Accepted by TMLR ()

Abstract: The Focal Loss is a widely deployed loss function that is used to train various types of deep learning models. It is a modification of the cross-entropy loss designed to mitigate the effect of class imbalance in dense object detection tasks. By downweighting the losses for easy, correctly classified samples, the method places more emphasis on harder, misclassified ones. As a result, gradient updates are not dominated by samples that the model already handles correctly. The downweighting of the loss is achieved by scaling the cross-entropy loss with a term that depends on a focusing parameter $\gamma$. In this paper, we highlight an unaddressed numerical instability of the Focal Loss that arises when this focusing parameter is set to a value between 0 and 1. We present the theoretical basis of this numerical instability, show that it can be detected in the computation of Focal Loss gradients, and demonstrate its effects across several classification and segmentation tasks. Additionally, we propose a straightforward modification to the original Focal Loss to ensure stability whenever these unstable focusing parameter values are used.

koshaakkkhiburrrniiveristty edu Published in Transactions on Machine Learning Research (11/2025)
A Note On The Stability Of The Focal Loss
Martijn P. van Leeuwen
mappanddeeernniibbbrrrrrrrrrrree edu Dept. of Intelligent Systems, Research Center for Cognitive Science and Artificial Intelligence, Tiburg School of Humonities and Digital Sciences, Tülburg University
Dept. of Intelligent Systems, Reseorch Center for Cognâtive Science and Artificial Intelligence, Tüburg School of Humonities and Digital Sciences, Tilburg University
Koen V. Haak
Dept. of Intellgent. Systems, Research Center for Cognátine Science and Artificial Intelligence, Tibburg School of Humanities and Digital Scienoes, Tilbury University
Görkem Saygili
g.sayyglli@tihhuuunnnnnnt.o edu
Eric Postma
C.0. c.o.postma Qobbrubrtbburrruueerrrstyyy edu Dept. of Computational Cognälive Science, Research Center for Cognitive Science and Artificial Intelligence, Tüburg School of Humanities and Digital Sciences, Tibbug University
L.L. Sharon Ong
Lt bl.onn@tibbrrgunierssty..cmm edu
Dept. of Intelligent Systems, Reseorch Center for Cognitive Science end Artificial Intelligence, Tilburg School of Humanities and Digital Sciences, Tilburg University
Reviewed on OpenReview: https: openreview. net/ permffrum? id- eCYActnGbu
Abstract
The Focal Loss is widely deployed loss function that is used to train various types of deep learning models. It is a modification of the cross-entropy loss designed to mitigate the effect of class imbalance in dense object detection lasks. By downweighting the losses for easy, correctly classified samples, the method places more emphasis on harder, misclassified ones. As a result, gradient updates are not dominated by samples that the model alrcady handles correctly. The downweighting of the loss is achieved by scaling the cross-entropy loss with term that depends a focusing parameter 7. Tn this paper, highlight unaddressed numerical instability of the Focal Loss that arises when this focusing parameter is set to a value between 0 and 1. We present the theoretical basis of this numerical instability, show that can be detected in the computation of Focal Loss gradients, and demonstrate its effects across several classification and sogmentation tasks. Additionally, we propose a straightforward modification to the original Focal Loss to ensure stability whenever these unstable focusing parameter values are used.
1 Introduction
The Focal Loss is widely used in one-stage object detectors, medical imaging, image segmentation, and pose estimation tasks. (Terven et al. 2023). This function modifies the distribution-based cross- entropy loss by introducing focusing parameter (7) that downweights the penalty that is applied to "casy" examples (Lin et al. (017). This downweighting of the loss prevents gradients from being dominated by easy examples, allowing the model to focus on difficult samples. (Lin el al. (017). This is particularly useful when the training data contains a large proportion of background (or other) class samples.
The selection of parameter 7, and with this, the extent which easy examples are downweighted, should be done via cross-validation (Lin al. 2017). Lin et al. (2077) reported that using y of 2 led to the

---

Published in Transactions on Machine Learning Research (11/2025)
best results in their experiments (Lin et al. 2017 There is, however, limit to what y values can be used. Selecting y values much larger than 2 has been shown to result in gradients close to 0 for relatively low model outputs, causing training to fail (Mukhoti et al. 2020). This paper will shed light on the other end of the spectrum, showing that the Focal Loss gradients can become unstable whenever y is too small. More specif- ically, we address a numerical instability of the Focal Loss that arises whenever a 7 is set to a value bet ween and These y values can cause the Focal Loss derivative to become undefined and destabilize model training due to a singularity. This singularity arises whenever the Focal Loss, in combination with a ? on the open unit interval, is used to learn a task for which model can confidently predict correct class label. We will demonstrate that this numerical instability of the Focal Loss is not only a theoretical problem by showing that a simple convolutional neural network (CNN), a vision transformer (VTT) (Wu et al. 2020), and a 2D U-net (Ronnbergger et al.) 2015) can return undefined loss values during training whenever values on the open unit interval are used. Henceforth, we will refer to the ry values between and 1 as unstable y values.
The original Focal Loss paper (Lin et al. 2017) did not address this numerical instability and presents experimental results generated with y between 0 and 1. While their findings show that training with unstable y values does not always lead to instability, our experiments highlight certain training conditions that increase the chance of encountering this instability.
For instance, will show that using unbalanced datasets increases the likelihood of encountering mu- merical instability when using unstable y values. Especially in the medical field, collected datasets are often dlas--imbllanced as a result of the prevalence of diseased patients (Salmi et al.] 2024). Given that the Focal Loss is designed to deal with class imbalance, using the Focal Loss in medical machine learning
tasks is not an uncommon approach (Ahmed et al. 2022 Romdhane & Pr 2020 Tran et al. 2019). Class imbalance is also encountered in other fields such as autonomous driving (Chen & Qin 2022 and fault detection (Zareaporr et al. 2021). Due to the class imbalance in these tasks, the Focal Loss is also employed in research on these topics (Wei ct al. 2022 Carranza-Garcia et al.) 2021). The optimal y value for any of these applications could be in the unstable range. To avoid numerical instability, it may be necessary to use a suboptimal value, which can in turn limit overall model performance. It is therefore crucial to address this instability, so that all possible y can be considered without risking training instability.
This instability is likely not confined to the Focal Loss itself but may also influence other loss func- tions that build upon or are drived from it. In recent years, various modifications of the Focal Loss have been proposed, such as the Generalized Focal Loss (Li et al. 2020), the Adaptive Focal Loss [Islam et al. 2024), and the Unified Focal Loss (Yeung el. al.) 2022). The Generalized Focal Loss extends the Focal Loss to jointly model classification confidence and localization quality in a joint representation, applying a focusing parameter similar to the original Focal Loss. In the Adaptive Focal Loss (Islam et al. 2024), the focusing parameter is dynamically adjusted during training, and the Unified Focal Loss combines the Focal Loss with the Focal Tversky Loss (Abraham & Khan 2019) but constrains the focusing parameter to fixed range between O and 1. All these new loss functions build upon the Focal Loss and similarly employ the focusing parameter as in the original formulation. As result, they are likely to exhibit the same numerical instability that we address in this paper.
In this paper, we highlight the origin of the instability and demonstrate how to resolve it. Without altering the original behavior of the Focal Loss, our proposed stabilization approach ensures compatibility with existing methods that employ the Focal Loss using these unstable 7 values. Our contribution is therefore not only to provide evidence that the instability exists, but also to apply minimally invasive modification to eliminate it, while simultaneously eliminating the instability in all Focal Loss variants that use the focusing parameter similarly.
2 Methods
This section will review the definition of the Focal Loss and its derivative to explain the origin of the instability. We then show that the numerical instability becomes apparent when computing the gradients of the original Focal Loss with unstable y values. Additionally, to show that this instability is not only
2

---

Published in Transactions on Machine Learning Research (11/2025)
theoretical obstacle, will demonstrate that under certain conditions, the instability can be induced in binary classification and 2D segmentation tasks. Lastly, we present modification of the original Focal Loss that eliminates the instability whenever unstable values are used.
2.1 Cross-entrowe and the Focal Loss
The Focal Loss was introduced to address class imbalance by reducing the effect of easily classifiable examples, thereby placing more emphasis on harder, misclassified ones {Lin et al. 2017p. This is achieved by modulating the standard cross-entropy loss in Equation (1) with a scaling factor that is based on the prediction error and a focusing parameter Ty. This modulating factor ensures that the smaller the prediction error becomes, the more the cross-entropy downscaled. In other words, predictions that are closer to the correct label (easy examples) are downscaled by the focusing parameter 7. Note that whenever a y of 0 is used, the Focal Loss simplifies to the cross-entropy loss. For simplicity, without loss of generality, cross-entropy loss function to the binary cross-entropy loss and reformulate the equations to foreground (G))) and background loss (Cag). For consistency, we make use of the same notation for the ground truth (y) and model output (p) as was used in the original Focal Loss paper (Lin et al. 2017). While the original Focal Loss paper reformulates the loss as a foreground loss for notational convenience, this paper explicitly highlights both the foreground and background components of the Focal Loss
will simplify the
CCE(y,P) = - log(p) - (1 y) log(1 - p)
(1)
Sto
Lbg
CF (F( = -4, (1 p)" log(p) (1 an) u) p² log(1 - p)
(2)
Ero
Cay
The Focal Loss, as defined in Equation 2 downscales both the foreground and background loss equally with the focusing parameter 7. As shown Equation g
the Focal Loss also includes a parameter a that can be used to scale the contribution of the foreground and the background loss relative each other.
2.2 Derivative the Focal Loss
Figure D and Figure D show the foreground and background components of the Focal Loss for different model outputs p when changing the value for the focusing parameter y. These plots show that an increase in 7 will cause the Focal Loss to show near-zero loss values for model outputs close to the ground truth label, consequently lowering their associated gradients. As previously decomposed the Focal Loss into
y=0
0=0.5
y=1
y=2
y=5
3.0
3.0
2.5
2.5
2.0
2.0
d.c./dpp
-1
d.ng/dp
1.5
1.5
1.0
1.0
-2
0.5
0.5
0.
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
p
p
p
P
(a)
(b)
(c)
(d)
Figure 1: Foreground (a) and background (b) components the Focal Loss and their associated foreground
(c) and background (d) derivatives. The losses and derivatives were caleulated with different 7 values and a fixed a of 0.5. The orange markers indicate undefined gradient values a result of division by in the derivative of the Focal Loss and represent the model outputs when p is equal to 1 in Figure (c) and 0 in Figure (d).
3

---

Published in Transactions on Machine Learning Research (11/2025)
foreground and background loss, can define the Focal Loss derivative the sum of the foreground
and background components, as shown in Equation (3) These two components are defined in Equations
4 and (5), and are displayed in Figure De and Figure Dd. The derivation of these equations is included in Appendix A.1
dLr(p:7,0.) (o + dLig 7,0))
(3)
dp
dp
dp
dL dL (p, as (7(1-p) -lgg)) (1-p)"
(4)
dp
p
dL 7,)) = -11 ai) aa) (1p~~1og(1 -100111100 -p) - p"
(5)
dp
1-P)
2.3 Focal Loss Instability
In the derivative of the Focal Loss, a (7 - 1) exponent is introduced in both the foreground and background components. Whenever 0 < < 1, this (7 - term becomes negative, and the model output raised to the power of a negative number, creating a fraction with the model output the denominator. An example of this illustrated in Equations for y 0.5.
6 and showing the derivatives of the foreground and background loss
dC (p)
(vl1-)
0.5
log(p) -T-P P
dp
7=03 = at
(VI-) -1
p
(6)
d. as) 1=0.5 = = ar))
0.5 log( 1 p)
vP
(7)
dp
1 p
The scenarios involving 7 value of 0.5, where p equals 1 for the foreground and 0 for the background component, shown in Equations (8) and (9) These equations show that at these points, the fraction introduced in the derivative leads to division by O, creating a singularity that causes training instability. Although in this example 7 was set to 0.5, this singularity holds for all y values between and 1, as all these values introduce a fraction in the Focal Loss derivative with the model output in the denominator.
dC (as) dp
1p=1,7=0.5 = ai (.0 0) = Undefinded
-0--
(8)
dLbg (as) 1=00,=0055 1== -(1 --1 (1-az) oz)
(00 vo -0)
Undefinded
(9)
dp
Consider training a binary classification model with the Focal Loss and a y value of 0.5, to distinguish a foreground class from a background class. When the foreground is easily separated from the background class, the model will quickly produce high model outputs for the correct classes. Whenever these model outputs are equal (within floating-poin precision) to the ground truth value (y = p), the derivative will become undefined due to a division by O, consequently triggering the instability. When faced with more challenging task, the model is unlikely to produce model outputs equal to the ground truth, thereby preventing instability from being triggered. Since most deep learning tasks are complex, this is presumably why the instability is not always an issue and why it has not yet. been addressed the literature.
One important note to consider when discussing the stability of the Focal Loss is that whenever the opposite of the ground truth is predicted by the model, log(0) is introduced in the equation of the Focal Loss, which also causes instabilitics. Note that the instability that this paper addresses occurs whenever the model produces output values that are equal to the ground truth. In other words, the log(0) instability occurs when the prediction error becomes extremely large, whereas the instability that we address occurs whenever the prediction crror becomes near-zero. Since machine learning models are trained to minimize prediction error, they are optimized towards a state where this instability will eventually occur if it is not addressed.
4

---

Published in Transactions on Machine Learning Research (11/2025)
y=0
y=
y=5
0.0
15
15
-0.2
dLg/dx
dEng/dx
0.4
10
10
-0.4
5
0.2
-0.6
0
-25 25
-2
0
25
-2
0
25
-25
25
x
X (c)
(a)
(b)
(d)
Figure Computed values for the foreground (a) and background (b) components of the Focal Loss in combination with the foreground (c) and background (d) gradients. These losses and gradients were com- puted with model outputs (x) that were not yet processed by the sigmoid and ranged between [-38,88] Both the losses and gradients were computed with the
_fcaal loss function with an a of 0.5, and different values for 7. The orange markers indicate the values for that cause "NaN" values when computing the loss gradients. They represents the model outpul when is equal to 1 in Figure (c) and 0 in Figure (d).
2.3.1 Numerical Gradient Computation
In this subsection, show that the instability occurs when computing the gradients with the original Focal Loss function implementation of the Focal Loss applies sigmoid activation function to the model outputs as shown in Equation (10,, in which the model output prior to applying the sigmoid is defined as æ. After the sigmoid, the Focal Loss calculatod using Equation 2 Figure 2 and Figure 2 show the foreground and background
focal_loss, used in the seminal work {Lin et al., 2017). This
components of the Focal Loss when computing them with the original Focal Loss function.
p p=o(x) Ite
(10)
Similar to Figure Figure shows the foreground and background loss combination with their associated gradients. Figure shows the Focal Loss for model outputs that are processed by a sigmoid, and Figure 2 shows the Focal Loss for unprocessed model outputs (æ)) These unprocossod output values
not confined to range of [0.1] but can theoretically span from [-),,00]. Figure 2 and Figure 2 present the computed Focal Loss gradients for the foreground and background components of the Focal Loss. The orange markers in indicatte the output values that cause the gradient to become undefined, consequently returning a 'NaN'. The points at which the loss becomes undefined indicate where instability arises the computation of the original Focal Loss.
2.4 Stabilized Focal Loss
As described in the previous section, the numerical instability of the Focal Loss arises as a result division by in its derivative. One commonly used approach to prevent a division by in loss function is the introduction of smoothing constant e in the denominator of the loss, which is stabilization method that is also applied to the well-known Dice loss (Milletari et al. 2016; Sudre et al. 2017). We propose modification of the original Focal Loss that leads to the introduction of smoothing constant in the denominator of its dcrivative when unstable values are used. This slightly differs from what is done to stabilize the Dice loss, in which the division by zero is prevented in the loss itself. Instead, we modily the original Focal Loss with a parameter so that the e term is placed in the denominator its derivative and not in the loss itself. This ultimetaly prevents division by zero when computing the gradient with unstable y values.
The modified Focal Loss and the derivatives for its foreground and background components de-

---

Published in Transactions on Machine Learning Research (11/2025)
fined as shown in Equations [1 12 and 13 By introducing constant denominator term independent of the model output, the proposed modification ensures numerical stability for values between 0 and 1. This prevents division by zero when the prediction approaches the ground truth within floating-point precision, effectively climinating the instability in the original Focal Loss. Note that smaller y values will cause the denominator to approach O more quickly for model outpuls close to the ground truth, compared to when larger value for y is used. This means that a larger required to ensure stability whenever smaller 7 value is used. We ran the experiments in this paper with a value of equal to le 3, as it stabilized model training whenever a as small 0.1 was used.
CFm P. C4, -ay (1 p 7) log(p) (1 - a-) 3) (p + c)? log(1 p)
(11)
Lto
as (1(L (7(1- -pl ->=p+e)---lllll (1 (1-P+) -1-++e)7 d. 7, a2)
(12)
dp
p
(p -- (c - p) (p +2)
(13)
dp
1
When revisiting the example in which is equal to 0.5, the derivatives for the foreground and background loss become equal to Equation (14h and 155.. When the model output is again equal to 1 in the foreground
and 0 in the background loss, as shown in Equation (16) and (17), the division by zero is prevented by the
smoothing constant 6. The implementation details of the modified version of the Focal Loss can be found Appendix A.3
dl (L l az
0.5
log(p) VI-P+c
(14)
dp
VT
p
P
dhig (p,
0.5
VD+E
7-0 1 -(1 - (11-ap) ar)
log(1 -p)
(15)
dp
vD
+
1
dL
m mm1 dLfg(atnE)
0
Ve
(16)
dp
Vf
1
d.bo
0
1p=0,-0.55- (--0
at)
oo)
(17)
dp
Ve
1
2.5 Experiments
To demonstrate the instability of the Focal Loss, we conducted several experiments. First, we tested whether the instability could occur when training basic convolutional neural network (CNN) (shown in Appendix A.4] to perform a binary classification task on the MNIST dataset (Deng) 2012). We then examined if this instability could be induced on a larger and more complex dataset, the C-FAR-10 dataset (Krizhevsky et al., 2009), using CNN and Vision Transformer (ViT) (Wu et al.] 2020). In a final experiment, we evaluated whether the instability could be observed during the training of a 2D U-Net (Ronneberger et al.) 2015 (implementation shown in A.6) will describe how these experiments were set up.
segmentation task using the MNIST dataset. This subsection
2.5.1 Binary Classification - MNIST
In the first experiment, we divided the MNIST dataset (Deng 2012) numbers ranging from to 9, into two classes, where threshold determined which numbers belonged to which class. The goal of the CNN was to learn to distinguish between the two classes by training for 100 epochs. As we are interested in the stability of the Focal Loss during training, we did not focus on model performance, but rather on whether the model could complete all 100 epochs without encountering any instabilities. In this experiment, we tested stable (0,1,2,3,4,5) and unstable y values (0 < < 1) to verify that the instability only occurs when using unstable y values.
dataset composed of handwritten
To transform the multiclass MNIST dataset into dataset that could be used for binary classifica-

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: The number of samples each class of the MNIST dataset (Dngg 2012) when using different class distributions. The class distribution was determined by binarization threshold ranging from The threshold value indicates the cut-off value for the classes belonging to class A or class B. All values larger than the threshold belonged to class B, and all classes below equal to the threshold belonged to class A.
Threshold Samples Class A
Samples Class B
Class A /B ratio
5.923
54.077
0.11
1
22.665
47.335
0.27
18.623
41.377
0.45
3
24.7541
35.246
0.70
4
30.596
29.404
1.04
5
36.017
23.983
1.50
6
41.935
18.065
2.32
7
48,200
11.800
4.08
8
54.051
5.949
9.09
/
2
3
4
S
6
7
9
(a)
6
/
2
3
7
8
9
(b)
3
(c)
Figure 3: Example imput images from the MNIST dataset (Deng 2012) for digits 0-9. The original images are shown in (a), while (b) and (c) illustrate the effects of adding "Medium Noise" and "High Noise," respectively. The 'Medium Noise' was generated by multiplying uniformly sampled noise between 0 and 1 by 0.5, while the "High Noise' was generated by multiplying this noise by 0.75.
tion lask, we applied a threshold the MNIST class labels, restructuring the dataset into a dataset with two classes: foreground (A) and background (B) class. Because the number of samples for each class of the MNIST training dataset is approximately the same (Hamidi & Borji 2010), changing this threshold allows for incrementally modifying the foreground-Dackgronndd ratio. Assessing the effect of changing this threshold provides insight into whether an imba lance in classes influences training stability whenever unstable y values are used. For example, a threshold of 4 means that the MNIST numbers with classes 0-4 belong to class A and the classes 5-9 belong to class B. Adjusting this threshold, therefore, changes the degree of class imbalance. By increasing the imbalance, we intentionally introduced bias toward the ma- jority class, resulting in more confident predictions for the correct labels, thereby aiming to induce instability.
The initial part of this experiment classification task influenced the expression of the instability. The second part of the experiment repeated the initial experiment, but trained the CNN with random noise added to the input images. By adding noise, we can evaluate whether increasing the classification task's difficulty mitigates the unstable behavior of the Focal Loss. The noise was generated by sampling random values from a uniform distribution over [0, 1). These values were then scaled by 0.5 and 0.75 to produce what we refer as "Medium Noisc" and "High Noise," " respectively. This randomly sampled noise was then added the input images. Examples of input images with and without added noise shown in Figure 3
designed test whether increasing class imbalance in the

---

Published in Transactions on Machine Learning Research (11/2025)
2.5.2 Binary Classification CIFAR-10
The CIFAR-10 dataset (Krizhevsky et al., 2009) is composed of images belonging to one of the following classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck. To perform binary classification on this dataset, the classes were partitioned into two classes: animals and vehicles. Each class of the original CIFAR-10 classes contained 5000 images, and with 6 animal and 4 vehicle classes, the classes for the restructured dataset. became slightly inbalanced (6:4 ratio). This setup, with its slight imbalance and the use of Focal Loss, reflects a realistic scenario commonly encountered in real-world datasets. Following dataset preparation, a Vision Transformer (ViT) (Wu et al.] 2020) and a CNN were trained to perform the binary classification task. The CNN was the same model that was used in the MNIST experiment, with some minor adjustments to account for the difference in input sizes. The used ViT (Wu et al. 2020) took 224x224 images as input and divided each image into 16x16 patches, before flattening them, and projecting each patch into a learnable embedding. More information about the used VIT can be found in Appendix A.5 Both models were trained for 1000 epochs with a batch size of 128 and a y and ag of 0.5 without using pre-trained weights. We trained on the complete training datasct, did not perform any hyperparamcter optimization or lesting.
2.5.3 2D Segmentation MNIST
In the final experiment, we lested whether the instability could be induced when training a model to segment the numbers in the MNIST dataset using a 2D U-net [Ronebberger et al. (2015). Since the MNIST dataset is intended for classification tasks, it does not include segmentation masks. We therefore applied a threshold of 0.5 to the imput images, setting all pixels that exceeded this value to the foreground of the segmentation mask and setting all pixels below this value to the background class. Similar to the binary classification task, we repeated the experiment after adding noise to the input data to test whether increasing the difficulty of the segmentation task influenced training stability. The noise was added after creating the segmentation masks to maintain consistent segmentation mask across experiments. After preprocessing of the data, the 2D U-net was trained with the Focal Loss using y and a of 0.5 for 1000 epochs. More details on the U-Net architecture can be found in Appendix A.6
3 Results
In this section, we demonstrate the Focal Loss instability that is caused by unstable y values. We report our findings for the original Focal Loss and also analyze how our modified version impacts training stability.
3.1 Binary Classification MNIST
For the binary classification task, we trained the CNN for 100 epochs with y values ranging from to 5 using the different class A/B ratios that are shown in Table [ Whenever a 'NaN" was encountered during training, training was stopped, otherwise, training would continue until all 100 epochs were completed. The results for these experiments are presented in Figure 4 where the number of completed epochs is shown for different and A/B class ratios. The left plot in Figure 4a shows that when was set to 0 or to values greater than 1, training remained stable across all class A/B ratios, completing all 100 epochs without any signs of instability.
The middle-column figure shows the results of training with Y values ranging from 0.1 to 0.9 in in- crements of 0.1 From this figure, we see that using 7 values between 0 and 1 frequently causes instability. These instabilities quickly arise, especially for smaller y values and unbalanced datasets. However, when using y of 0.9 in combination with a relatively balanced dataset, all 100 epochs were completed. It is, however, not unlikely that whenever these models were trained for more than 100 epochs, the instability would still have been found in a later epoch.
Figure 4 and 4 show that adding noise has a mitigating effect on how quickly the instabilities are detected. These figures indicate that higher levels of noise result in a greater number of y values exhibiting stable behavior. Moreover, introducing noise enables wider range of class distributions to achieve stable
8

---

Published in Transactions on Machine Learning Research (11/2025)
training outcomes, particularly for larger y values. Furthermore, the left plot in Figure A and Figure + also show that adding noise did not affect the number of completed epochs when training with stable y values.
After stabilizing the Focal Loss with the smoothing constant, all experiments were repeated, for
Completed
Completed
Unstable/Stable Focal loss
Unstable Focal loss
Stable Focal loss
100
100
100
9.09
9.09
9.09
60
Completed Epochs
4.08
80
4,08
B0
Completed Epochs
ratta
class A/B rati
2.32
2.32
60
60
1.04
Completed
40
0.7
40
0.45
0.45
0.45
0.27
20
0.27
20
0.27
20
0.11
0.11
0.11
2
5
0.10.20.30.40..3 0...0.0....................
(a) No noise
x
Not Completed
Completed
Unstable/Stable Focal loss
Unstable Focal loss
Stable Focal loss
100
9.09
9.09
X
X
X
X
X
x
9.09
4.08
2.32
80
Completed Epochs
class A/B rabi
4.08
2.32
X
x
X
X
X
X
X
X
80
Congleted Epochs
A/B rabi
4.08
2.32
80
Completed Epochs
1.5
1.5
x
1.5
1.04
1.04
1.04
40
400
40
0.45
20
0.27
0.27
20
0.11
0.11
0.10.2..0.44055.666.0..0.3
0.10.20.1.440.0.66
7.8
(b) Medium Noise
X
Not Completed
Completed
Unstable/Stable Focal loss
Unstable Focal loss
Stable Focal loss
9.09
9.09
X
X
X
X
X
9.09
4.08
80
Completed Epochs
4,08
X
X
X
80
Completed Epochs
4.08
80
Completed Epochs
6935 A/B raro
2.32
class A/B rabo
2.32
2.32
1.5
1.5
1.5
1.04
A/B
1.04
0.7
40
40
40
0.45
20
0.27
20
0.27
20
0.11
0.11
0.11
5
110.20.30.40..5066..0.....
0.10.20.30.40..
7..809
(c) High Noise
Figure 4: (a): Binary classification results on the MNIST dataset, illustrating the number of completed epochs for different values under varying class A/B ratios. The left plots show model results using values of 0,1,2,3,4, and 5. The middle and left plots show the number of completed epochs whenever the values of 0.1 to 0.9 with increments of 0.1 are used, where the middle plot shows the results when training with the original Focal Loss, and the plot on the right shows the results when using the stabilized Focal Loss. Each plot in the figures includes a marker that indicates whether all 100 epochs were completed. (b-c): Experiment results when the initial experiment. is repeated with 'Medium" and 'High' amounts of noise added to the input images to increase the difficulty of the classification task.
9

---

Published in Transactions on Machine Learning Research (11/2025)
which the results are shown the right in Figure These results show that the modification of the Focal Loss successfully eliminated the instability, no more instabilities are reported in any of the experiments.
3.2 Binary Classification CIFAR-10
Figure 5 shows the Focal Loss values when training the CNN and VIT to perform the binary classification tasks on the CIFAR-10 dataset. Similar to the classification results on the MNIST dataset, Figure 5h shows that the numerical instability occurs during training for both models. Figure 5 shows the results when the models were trained with the stabilized Focal Loss. Similar to the MNIST classification experiments, no further instabilities were reported when the models were trained with the stabilized Focal Loss, and both models completed training of all 1000 epochs.
3.3 2D Segmentation MNIST
The results of training the 2D U-nct are shown in Figure 6 This figure reports the computed Focal Loss for each epoch, but halted training whenever a 'NaN' was encountered or all 1000 epochs were completed. When
CNN
VT
Unstable Focal Loss
Stable Focal Loss
0.30
0.30
0.25
0.25
Focal loss
0.20
Focal loss
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
200
400
600
800
1000
200
400
600
800
1000
Epoch (a)
Epoch (b)
Figure Results after training the CNN and VIT with the Focal Loss using y and or of 0.5 for 1000 epochs on the CIFAR-10 dataset with the original Focal Loss (a), and the stabilized Focal Loss (b). A cross indicates the epoch at which "NaN" was encountered during training.
No Noise
Medium Noise
High Noise
Unstable Focal Loss
Stable Focal Loss
0.03
0.03
Focal loss
Focal loss
0.02
0.02
0.01
0.01
0.00
0.00
200
400
600
800
1000
200
400
600
800
1000
Epoch
Epoch
(a)
(b)
Figure Results after training the 2D U-net with the Focal Loss using a 7 and a₂ of 0.5 for 1000 epochs on the MNIST dataset with the original Focal Loss (a), and the stabilized Focal Loss (b). The model was trained without noise, and with 'Medium' and 'High" noise levels. A cross indicates the epoch at which "NaN" was encountered during training.
10

---

Published in Transactions on Machine Learning Research (11/2025)
noise was added to the imput data, instability was quickly observed. Introducing small amount of noise delayed its onset, and at higher noise levels, instability was no longer observed. The segmentation results are consistent with the results from the binary classification task. Again, when using the modified version of the Focal Loss, no more instabilitics were reported, and all three models were trained to completion. Note that the losses have different asymptoles, which originale from the way that the noise was added and the segmentation masks were generatod. Introducing noise before generating the masks may have reduced the clarity of the boundary between foreground and background. Consequently, accurately determining this boundary becomes more difficult, leading to increase in loss and corresponding shift in the asymptote
4 Discussion and Conclusion
This paper addresses a hitherto unreported instability of the Focal Loss when the focussing parameter yy is set to a value between 0 and 1. We showed that this instability is not only mathematically derivable but can also be demonstrated using some simple deep learning experiments. Due to the singularity that arises in the derivative of the Focal Loss when using these unstable y values, training deep learning models like basic CNN, ViT, or 2D U-net can lead to unstable behavior. Our experiments suggest that when training with the Focal Loss using unstable y values, datasets with severe class imbalance are more prone to exhibiting instabilities earlier than when using balanced datasets, and that task complexity further influences the rate at which these instabilities manifest. A likely explanation is that models trained on easy tasks tend to overfit quickly, producing highly confident prodictions. As a result, the model outputs become equal to the true class values, which leads to a singularity in the derivative of the Focal Loss consequently causing instability. The more difficult the task at hand, the more epochs are needed to reach a state where the predictions are confident enough to trigger the instability, This is also demonstrated in our experiments, which show that increasing task difficulty by adding noise delays the onset of the instability. With the presented experiments, we highlight that the instability is not necessarily an issue in all training scenarios, but that it can arise under certain conditions.
To resolve this instability, we proposed modification of the original Focal Loss by adding a smoothing constant to the term that downscales the cross-entropy loss. This ensures that the singularity in the Focal Loss derivative is climinated, consequently stabilizing model training. While unstable y values triggered instability with the original Focal Loss in our experiments, training with the modified version led to completing all epochs in each experiment.
In this paper, we provide numerical and experimental evidence of the existence of this Focal Loss instability. Our experiments highlight that the instability can be induced when training deep learning models. We therefore recommend refraining from using y values that fall between and when using the original Focal Loss. If by design, the only possible values for y fall between 0 and as is the case for the Unified Focal Loss (Yeung et. al. 2022 we recommend using our stabilized version of the Focal Loss to eliminate the chance of encountering instability, Yeung et al. (2022) did not report that their method suffered from numerical instabilities, even though their loss makes use of these unstable 7 values. Their published code shows clipping of the model outputs, which would prevent their model from reaching model outputs that trigger the instability. However, no explanations were provided for the clipping operation. Additionally, their paper focuses on complex segmentation tasks, which, as we have shown, are less prone to expressing instability. It could be possible that applying the Unified Focal Loss to more simplistic segmentation tasks could still trigger the instability.
In summary, we identified an unaddressed numerical instability in the Focal Loss and proposed the addition of a stabilizing smoothing constant to prevent it from destabilizing model training. Our experi- ments showed that after the addition of this smoothing constant to the Focal Loss, the instabilities were effectively removed. We therefore recommend either refraining from using the unstable y values when using the Focal Loss adopting our modification to prevent. instabilities from occurring.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Acknowledgements
This research has been funded by NWO Starter Grant (Project Reference: 510.023.062). Additionally, KVH gratefully acknowledges funding from the Dutch Research Council (NWO, Vidi grant no. 09150171910043)
References
Nabila Abraham and Naimul Mefraz Khan. A novel focal tversky loss function with improved attention u-net for lesion segmentation. In 2019 IEEE 16th international symposium biomedical imaging (ISBI 2019), pp. 683 687. IEEE, 2019.
Syed Rakin Ahmed, Andreanne Lemay, Katharina V Hoebel, and Jayashree Kalpathy-Cramer Focal loss improves repeatability of deep learning models. In Medical Imaging with Deep Learning, 2022.
Manuel Carranza- Garcia, Pedro Lara-Beníted Jorge Garcia- Garcia-Gutiérze Gutiérrez, and José C Riquelme. Enhancing object detection for autonomous driving by optimizing anchor generation and addressing class imbalance. Newrocomputingg 449:229 244, 2021.
Guancheng Chen and Huabiao Qin. dlsriniisceeeinative focal loss for extreme imbalanced multiclass object detection towards autonomous driving. The Visual Computer, 38(3):1051-1063 2022.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-1422 2012.
Mandana Hamidi and Ali Borji. Invariance analysis of modified c2 features: case study handwritten digit recognition. Machine Vision and Applications, 21:969 979, 2010.
Md Rakibul Islam, Riad Hassan, Abdullah Nazib, Kien Nguyen, Clinton Fookes, and Md Zahidul Is- lam. Enhancing semantic segmentation with adaptive focal loss: A novel approach. arXiv preprint arXi::2407.08828 2024.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in neural information processing systems, 33:21002-21011 2020.
Tsung-Yt Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image sogmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-57 571. Icee, 2016.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in neural information processing systems, 33: 15288 15299, 2020.
Taissir Fekih Romdhane and Mohamed Atri Pr. Electrocardiogram heartbeat classification based on deep convolutional neural network and focal loss. Computers in Biology and Medicine, 123:103866, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and compuler-assisted internention 2015: 18th inter- national conference, Munich, Germany, October 5-9, 2015, proceedings, part 111 18, pp. 234 241. Springer, 2015.
Mabrouka Salmi, Dalia Atif, Diego Oliva, Ajith Abraham, and Sebastian Ventura. Handling imbalanced medical datasets: review of decade of research. Artificial intelligence review, 57(10):273, 2024.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec Cily, QC, Canada, Seplember 14, Proceedings 3, pp. 240 248. Springer, 2017.
Juan Terven, Diana M Sordova-Esparra Alfonso Ramirez-Pedrawa,
Pedraza, Edgar A Chavez-Urbiola, and Julio A Hhoeroo-Gozallee Loss functions and metrics in deep learning. arXiv preprint (XX::23377006944 2023.
Giang Son Tran, Thi Phuong Nghiem, Van Thi Nguyen, Chi Mai Luong, and Jean-Christophe Burie. Tm- proving accuracy of lung nodule classification using deep learning with focal loss. Journal of healtheare engineering, 2019(1)5515416, 2019.
Xiao-Li Wei, Chun-Xia Zhang, Sang-Woon Woon Kim, Kai-Li Jing, Yong-Jun Wang, Shuang Xu, and Zhuang- Zhuang Xie. Seismic fault detection using convolutional neural networks with focal loss. Computers & Geosciences, 158:104968, 2022.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda, Visual transformers: Token-basod image representation and processing for computer vision, 2020.
Michael Yeung, Evis Sala, Marola-Bibiann Schönlieb, and Leonardo Rundo. Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Computerized Medical Imaging and Graphics, 95:102026, 2022.
Masoumeh Zareapoor, Pourya Shamsolmoali, and Jie Yang. Oversampling adversarial network for class- imbalanced fault diagnosis. Mechanical Systems and Signal Processing, 149:107175, 2021.
13

---

Published in Transactions on Machine Learning Research (11/2025)
A
Appendix
A.1
Derivation Focal Loss Derivative
This Appendix contains the derivations of the Focal loss derivative. We provide separate derivations for the foreground and background classes.
A.1.1
Complete Focal loss
(F(y,D.,,a))= = any (1 p)" log(p)
(1 at) (1-1)p" log(1 p)
(18)
Cza
Cbg
A.1.2
Foreground Derivative
dL (C.g(P,7,a1)
d(1 - p)"
log(p) + dllo((p)
(1 -p)7
y=1 -04
dp
dp
dp
( p)> log(p) (1-p)) p)7
(19)
p
an (111p)>-11og) 1og(p)))))))))))))) (1 (1 -p)"
p
A.2 Background Derivative
dh (p. 7, a.)
dp" dp
dlog(1 - p)
w= = =-11 -(1-0,)
log(1 - -p) +
p²
dp
dp
(20)
p"
= =- - az))
(pp>- log(1 -p) 1-
14

---

Published in Transactions on Machine Learning Research (11/2025)
A.3 Modified Focal Loss
In this Appendix, show the Python implementation of the modified version of the original Focal loss et al. 2017p. Modifications to the original code are indicated by the "#Modification' comment.
Lin
import torch
import worch.an.functiene
from torchvision. utils import _og__ppi_uagee_om
#Modefication
def sigmoid_focal (oddifid)
imputa: torch. Tensor, targets: torch. Tensor,
10
alpha: float 0.25, gamma: float 2, reduction: str "none", epsilon=ie-3 #Modification
11
12
13
14
16
-> torch. Tensor:
17
Modified version of the Facal Loss. The epsilon scalar that is added to the output stabslizes the model trasning. Whenever epaslon is set to O, it ssmplefies to the original Pocal loas.
Args:
inputs (Tensor): float tensor arbstrary shape. predsctions for each enample. targets (Tensor): float tensor with the shape inputs. Stores the bénary classification label for element en inputs for the negative class and for the posstive class).
26
27
28
29
alpha (float): Heightsng factor in range (0,1) balance posstáve us negative ezanples or for ignore. Default: "0.255
30
31
32
gamma (float); Exponent of the modulating factor p_ to batance easy us hard ezasples. Default: "22" (floot)))) Smoothing constant preventing the instabilåtães when gamma values between 0 and are used. Defauls: "'ee-3"
reductson (string): "'none'"" '''menn"" / 'sum!
"none" : No reduction will be applsed to the output. mean ".; The output will be averaged.
"sum!"": output will summed.
Default: "" ""one""..
Returns:
Loss tensor the reduction option applied.
44
46
#Hodiftcation of Original implementation from
46
https: /ttp//gttttttttg//egg con http//acbboorreaarchh//oaa wwwpeearrrcrr/rreeeeeeeeeeeeeeee wttp:///woooo////aaeeeeeeeeeeeee http:///bbob////////////e/.o wttp://wwwwee/ooaaa/eaaeeeeceeee
47
48
if not torch.jit. jit. jit..in_crriptin(() and not torch. jit. is. itsiiittaaaaa se_trrcingg):
40
_log_api_ hog_api_usage__ once
(oss_nodifieed #Hodification
50
torch. torch.sigaoid iiggoid(innputa
ce ce_loss F. binary_cross, crwss_entropy......eppep with logghhooogtttttnnuut targets, reduction="none"
(p) targets (1 p) * (1 targets)
loss coss ((1 p_trepsilon) ** ganma) #Modsficatson
if alpha >- 0:
alpha_t alpha targets alpha) targets) loss alpha_t loss
Check reductson option return loss accordingly reduction "none":
61
62
pass
15

---

Published in Transactions on Machine Learning Research (11/2025)
elif reduction "mean": loss loss. m) elif reduction "sum":
loss. (uss.sum()
67
else:
68
raise ValueError(
69
I'Invalid Value for arg 'reduction's "(reduction) An
70
Supported reduction modes: 'none', 'mean', 'aum
71
72
return losa
78
16

---

Published in Transactions on Machine Learning Research (11/2025)
A.4 CNN for Binary and Multiclass classification
A summary of the CNN (modified from CNN in Pytorch Tutorial https://pytorch.. org/tutorials/ Degiimerr/littz/iffar///// _aarrotttttrill._t html) that was used for binary classification is shown in Table 3 A code snippet that displays the implementation of this model in Python is also provided below.
Table 2: Summary of the binary classification CNN when using batch size of 64 to train on the MNIST dataset
Layer (lype:depthhiddx)
Output Shape
Param #
CNN
[64, 1]
Couv2d: 1-1
[64, 6, 24, 24]
156
MaxPool2d: 1-2
[64, 6, 12, 12]
Conv2d: 1-3
[64, 16, 8]
2.416
MaxPool2d: 1-4
[64, 16, 4, 4]
Linear: 1-5
664, 120
30.840
Linear: 1-6
664, 84]
10.164
Linear: 1-7
[64, 1]
85
Table 3: Summary of the binary classification CNN when using a batch size of 128 to train on the CIFAR-10 dataset
Layer (lyperdepth-iddx)
Output Shape
Param #
CNN
[128, 3]
Conv2d: 1-1
[128, 6, 28, 28]
156
MaxPool2d: 1-2
[128, 6, 14, 14]
Conv2d: 1-3
[128, 16, 10, 10]
2.416
MaxPool2d: 1-4
[128, 16, 5]
Linear:
[128, 120
48.120
Linear: 1-6
128, 84
10.164
Linear: 1-7
[128, 1]
import
import sorch.n..fucctonall
import torchinfo import summary
class (MMnnmModule)
def _ielft__ssefff____ sceef,no_cee co.esssee-=:) super()...init___]
self. self.channee chanmele--haamelee
10
Conv_output, paa__output_eize-int((( (((ip(tttttttttiiiiiieeeaa
11
self celv. ma Conv 6, 5)
self pool = MaxPool2d 2)
self conv2 nn. (6nv2d 16, 5)
self fcl Linear (itevarCCovv_utttt wwC.://oupput.eeeeeroooooooopppp
120)
self.fc2 Linear 84)
self fc3 Linear(84, mo_classes)
def forward x):
18
self (F. relu(self. wwt.:/eee.....c(eeoo (x)))
20
self self (F. relu(sell. come)())))
21
torch.flatten
22
23
F. F.relu(eeff.fell
24
F.relu(eelf.fe265
25
fc3))
26
return x
17

---

Published in Transactions on Machine Learning Research (11/2025)
A.5 Architecture Vision Transformer (ViT)
Summary of the Vision Transformer architecture. More details on this architecture can be found at https:
//huggingface Btaugginnface..cggooggeeoo poo.le/eiii-bba-eaaatceeeee
Table 4: Summary of the Vision Transformer architecture
Layer (typerdepth-idx)
Output Shape
Param #
Vision Transformer
152,064
PatchEmbed: 1-1
Conv2d:
590,592
Identily: 2-2
Dropout: 1-2
Identity: 1-3
Identity: 1-4
Sequential: 1-5
Block: 2-x (repeated 14 times)
LayerNorm
1,536
Attention
2,362,368
Identity
Identity
LayerNorm
1,536
Mlp
4,722,432
Identity
Identity
LayerNorm:
1,536
Identity: 1-7
Dropout: 1-8
Linear: 1-9
769
Total params:
85,799,425
Trainable params::
85,799,425
18

---

Published in Transactions on Machine Learning Research (11/2025)
A.6
U-Net
The U-Net Ronneberger et al.
2015 implementation used in this paper was adapted from
com/ http://lllaaaneeee////eeeee wttptetett/eeeeeeeeeeeeeeeteeett htaet///lo/mmmeerrr/r// master unet/unet. py with some minor modifications.
Table 5: Summary of the 2D U-Net architecture when using batch size of 64
Layer (type:depth-idx)
Output Shape
Param #
UNet
64, 1, 28, 28
Conv2d: 3-1
64, 64, 28, 28
640
BatchNorm2d: 3-2
64, 64, 28, 28
128
Conv2d: 3-4
64, 64, 28, 28
36,928
BatchNorm22: 3-5
64. 64, 28, 28
128
Sequential: 3-7
61. 128. 14, 14]
221,952
Sequential: 3-8
Sequential: 3-9
64, 512,
3,542,016
Sequential: 3-10
64, 1024, 1]
14,161,920
Con/Transpos:2d: 3-11
64, 512,
2,097.664
Sequential: 3-12
64, 512.1
7, 7,080.960
ConvTranspooe2d: 3-13
524,511
Sequential: 3-14
64, 256, 7.
1,771,008
Con-Transsos:2:::
Sequential: 3-16
64, 128, 14, 14
443,130
Tannpraaspps:::::: 3-17
64, 64, 28, 28
32,832
Sequential: 3-18
64, 64. 28, 28
110,976
Conv2d: 1-10
04. 1, 28, 28)
65
Total params
31, 31,042,369
Trainable params
31,042.309 0
Non-trainable params
Total mult-adds (G)
36.16
Input size (MB)
0.20
Norward/backwart size (MB)
425.34
Params (MB)
Estimated Total Sixe (MB)
549.71
#Dounloaded and modsffed from:
#https: wht.tps://gtthib http:///thhub..oo.lceeeeeeeccc http://www.......oooooooooeea/aa
import torch
from torch import an
import torch.an. ww.ch.an.fuueeee fun as
class DoubleConv(nn (nn Module):
def (self, in_ch, out_ch):
super (DoubleCoo,, selt)...init__..
self conv Sequuntiaiil
Conv2d out_ch, kernel_size=3, padding=1),
15
Conv2d (out, out_ch, hernel_size=3, padding-1),
16
Batchhorrm2a((utt_ch))) RatD((inllacerTrrr))
17
18
19
20
def (sru, x):
21
conv (x) return x
22
28
24
clase Module):
def (sel_ in_ch, out_ch):
super (Up, self up_scale nn Con/Taanpooe22iinncch out_ch, kezeel_ stride=2)
def (srl, x2):
x2 self (x2
33
diffY size [2] x2.size()
19

---

Published in Transactions on Machine Learning Research (11/2025)
diffX xi.size [3] x2. aize
(x2 [diffx 2, diffX diffX
diffY 2, diffY diffY //
98
carch. cat(c (xx2, x1], dim=1)
39
return x
40
41
42
class DownLayer (nn Nodule):
48
def ..init___ (self, in_ch, out_ch):
44
super (DownLaye, self) _eltt_..init_..)
self self.pool MaxPool2d(2, stride=2, padding-0) conv (in_ch, out_ch)
def (eel, x):
self conv wweeeeeffpoo((1)) pool (x))
return
53
clase UpLayer Module):
def initt._. (self, in in_ch, ch)
56
super (UpLayer, self)...init__. self up Up(in_ch, out_ch)
58
57
self
conv conv.conv
DoubleConv (in_ch,, out_ch)
58
50
def forward(self, x1, x
60
= self self.up(x1, x2)
61
= self conv (a)
return x
class Ullt (nn. Module)
def ___ni____ (self, cchnnees=1,, dimensions-1):
super(UNet, self) colf).__init__()
mublleCovv(chaneel: 64)
delf. down1 (6s 128)
70
delf.
down2 DounLayer 256)
delf.
Down
72
self, down4 DSwnlayer(512, 1024)
78
upl
UpLayer 512)
74
self. up2 U512 256)
75
self up3 UpLayer (256, 128)
76
self. up4 U1Layer(128, 64)
77
self last_conv nn. (64, dimensions, 1)
78
def forward(self, x):
x1 self.convi (x)
x2 self.downl (x1)
x3 self down.don2222 (x2)
x4 self down3do3333333
x5 self self.dowmd (x4)
x1 d_up self ulff.up1(e (x4, x5)
x2_up self up2(x3, xl_up)
x3_up up self uxx x2_up)
x4_up up ued1, x3_up)
59
output self. last__conv(x__ug
90
return output
91
20


Reviews:
[{"review_id": "iTL8WmT5V5", "reviewer": "Reviewer_Zb67", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper has a simple toy experiment to showcase the instability in practice and shows that the \"difficulty\" of the problem is critical. It only shows up in \"easy\" problems.\n\n- The paper's flow is relatitvely easy to follow.\n\n\n**Weaknesses**\n- (Major and fundamental) Incorrect derivation that is fundamental to the paper's claims. The paper claims in (8) and (9) that there is a division by zero in the limit. But this is not true (also, there seems to be notational inconsistency since the limit is for $\\hat{y}$ but I'm assuming it is really for $p$. The limit needs to use L'hopitals rule since both the numerator  $\\log(1-p)$ and denominator $\\sqrt{p}$ approach 0 as $p$ approaches 0. Taking the derivative of both we have $\\frac{1}{1-p}$ and $\\frac{1}{\\sqrt{p}}$ so the fraction becomes $\\frac{\\sqrt{p}}{1-p}$, which then simplifies to simply 0 instead of $\\frac{0}{0}$ as is claimed.\n\n\n- It is unclear if this is actually a practical problem or not given the problem above. I'm not convinced this is even a fundamental problem. Perhaps, it is a numerical instability problem but it should be addressed by doing numerically stable calculations rather than by modifying the Focal loss function itself to be different.k\n\n- The experiments are very simple and may suggest that indeed this numerical instability does not happen in practice since you have to have very extreme values before it causes numerical issues (38 or -38 is super high and probably just a numerical issue, not a fundamental instability).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper first attempts to show that their is a mathematic instability of the gradients of the focal loss when $\\gamma$ is between 0 and 1.\nThis results in a divide by $p$ raised to a fractional power and thus the possibility of divide by 0.\nThe paper claims (incorrectly) that in the limit the gradients become undefined when the classifier is super confident.\nThe paper proposes to add an positive epsilon to the denominator to avoid the divide by zero error.\nThen the paper showcases the instability in a simple MNIST experiment where the class imbalance and the difficulty of the problem is carefully controlled.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Ultimately, I believe the instability observed is NOT a mathematic instability but rather a numerical instability. Thus, the fix should not be to modify the loss function but to compute the loss function in a numerically stable way. This should be the focus of the paper, not modifying the loss function. Please look into numerically stable ways to compute the focal loss function.\n\n- Also, you need to completely rewrite the story if this is only a numerical instability, which I believe it is. The problem only showcases itself for extremely small values close to 0 or 1.\n\n- (Less critical but still important) Showcase a more realistic example where this instability causes problems. Without this, the motivation or relevance to the ML audience is questionable. I think you could possibly show it with a very small $\\gamma$ like 0.01 or 0.0001 even on a somewhat more realistic dataset. For example, the experiments in Unified Focal Loss that used clipping. If you can show that their results do not hold without clipping, then you have a strong case.\n\n- (Ideally) You would show that the numerically stable version is within floating-point precision.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2UmLgLJZzH", "reviewer": "Reviewer_Erbc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper is clearly written and easy to follow.\n2. Training to stabilize focal loss is an area of interest in optimization.\n3. The authors identifies and analyses the instability of the Focal Loss function from the perspectives of numerical instability, which for me is a quite interesting perspective.\n4. The proposed method is simple and effective, which can be easily implemented in practice.\n\n** Weakness **\n1. This instability is caused by the denominator approaches 0 when predicted probability converges to the true label. While the proposed solution (adding a small constant to the denominator to prevent division by zero) is indeed effective, it is a well-established technique in numerical computation for addressing singularities. From this aspect, the contribution may be viewed as more of a practical fix than an academic advancement.\n\n2. The authors have only demonstrate the instability with $\\gamma = 0.5$. The authors could shape such a conclusion via a more rigorous theoretical analysis.\n\n3. No related works are discussed. Several existing studies may also impact the stability of Focal Loss, and the authors should analyze and compare their method with such works. Relevant references include:\n\n    [1] Islam, Md Rakibul, et al. \"Enhancing Semantic Segmentation with Adaptive Focal Loss: A Novel Approach.\" arXiv preprint arXiv:2407.09828 (2024).\n\n    [2] Li, Xiang, et al. \"Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.\" Advances in neural information processing systems 33 (2020): 21002-21012.\n\n\n4. The empirical results are not sufficient to support the effectiveness of the proposed method. The authors only provide results on MNIST dataset, which is a very simple dataset. For now, this is not sufficiently comprehensive for the current standards within the deep learning community. The authors should provide more comprehensive experiments on more complex datasets to demonstrate the effectiveness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies and analyzes the instability of the Focal Loss function. The authors demonstrate that this instability arises when the focusing parameter $\\gamma$ is set between 0 and 1 (0.5 is used as the example), which can lead to numerical issues during model training. Specifically, the denominator in the loss function may approach zero as the predicted probability converges to the true label. To address this, the authors propose a simple yet effective modification to the Focal Loss by introducing a small constant $\\epsilon$, a common technique in numerical computation to avoid 0 denominator. Experimental results on binary classification and segmentation tasks using the MNIST dataset demonstrate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "I have not found any discussions about the limitations and potential negative societal impact. But in my opinion, this may not be a problem, since the work only focuses on the optimization in deep learning. Still, it is highly encouraged to add corresponding discussions.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wMS3NUdMIK", "reviewer": "Reviewer_EoP2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe authors consider a very specific and well defined problem, and do overall do a good job at both defining and addressing it. The theoretical analysis is sound and is well-described. Empirically, the authors consider both classification and segmentation experiments, which is good practice. This paper would likely have signifiance to practitions who frequently use the focal loss.\n\n**Weaknessess**\n\nThe primary weakness in my view is that the experiments are quite small scale. It's difficult to assess whether the results would extend beyond a simple setting like MNIST. I would encourage the authors to at least experiment on CIFAR-10 or ideally an ImageNet-scale dataset, although I understand that some careful tuning may be required to observe this instability on larger datasets. This would help establish the empirical significance of this work.\n\nA more minor concern is on Figures 1 and 2. These figures aims to show a qualitivative difference in the gradients between a gamma value of 0.5 and the other gamma values, which is central to the argument of why focal loss fails in this particular regime. Unfortunately, neither figure explicitly depicts a discontinuity at the point where instability ocurrs (presumably p=1.0 in figure 1c for example). For Figure 1cd, I would encourage including an empty circle or an x to indicate when the gradient is undefined for the orange curve. For Figure 2, I would similarly indicate when the gradient is undefined: all this figure appears to show now is when the gradients are undefined for the opposite reason to what the authors consider (namely, when the model is highly incorrect).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate a feature of the focal loss where for gamma parameter values between 0 and 1, the focal loss becomes unstable. Theoretically, the authors show that this instability occurs when the model output is very close to correct, which results in a division by zero in the computation of the model output gradient. The authors demonstrate this effect empirically and also propose a novel modification to the focal loss to avoid this instability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**\n\nPlease include experiments on a larger dataset (such as CIFAR or ideally ImageNet)\n\nPlease modify Figures 1 and 2 as described above", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)