“SECTION 3Focal Loss
The Focal Loss is designed to address the one-stage object detection scenario in which there is an extreme imbalance between foreground and background classes during training (e.g., 1:1000). We introduce the focal loss starting from the cross entropy (CE) loss for binary classification1:
CE(p,y)={−log§−log(1−p)if y=1otherwise.(1)
View SourceRight-click on figure for MathML and additional features.In the above y∈{±1} specifies the ground-truth class and p∈[0,1] is the model’s estimated probability for the class with label y=1. For notational convenience, we define pt:
pt={p1−pif y=1otherwise,(2)
View SourceRight-click on figure for MathML and additional features.and rewrite CE(p,y)=CE(pt)=−log(pt).

The CE loss can be seen as the blue (top) curve in Fig. 1. One notable property of this loss, which can be easily seen in its plot, is that even examples that are easily classified (pt≫.5) incur a loss with non-trivial magnitude. When summed over a large number of easy examples, these small loss values can overwhelm the rare class.

3.1 Balanced Cross Entropy
A common method for addressing class imbalance is to introduce a weighting factor α∈[0,1] for class 1 and 1−α for class −1. In practice α may be set by inverse class frequency or treated as a hyperparameter to set by cross validation. For notational convenience, we define αt analogously to how we defined pt. We write the α-balanced CE loss as:
CE(pt)=−αtlog(pt).(3)
View SourceRight-click on figure for MathML and additional features.This loss is a simple extension to CE that we consider as an experimental baseline for our proposed focal loss.

3.2 Focal Loss Definition
As our experiments will show, the large class imbalance encountered during training of dense detectors overwhelms the cross entropy loss. Easily classified negatives comprise the majority of the loss and dominate the gradient. While α balances the importance of positive/negative examples, it does not differentiate between easy/hard examples. Instead, we propose to reshape the loss function to down-weight easy examples and thus focus training on hard negatives.

More formally, we propose to add a modulating factor (1−pt)γ to the cross entropy loss, with tunable focusing parameter γ≥0. We define the focal loss as:
FL(pt)=−(1−pt)γlog(pt).(4)
View SourceRight-click on figure for MathML and additional features.The focal loss is visualized for several values of γ∈[0,5] in Fig. 1. We note two properties of the focal loss. (1) When an example is misclassified and pt is small, the modulating factor is near 1 and the loss is unaffected. As pt→1, the factor goes to 0 and the loss for well-classified examples is down-weighted. (2) The focusing parameter γ smoothly adjusts the rate at which easy examples are down-weighted. When γ=0, FL is equivalent to CE, and as γ is increased the effect of the modulating factor is likewise increased (we found γ=2 to work best in our experiments).

Intuitively, the modulating factor reduces the loss contribution from easy examples and extends the range in which an example receives low loss. For instance, with γ=2, an example classified with pt=0.9 would have 100× lower loss compared with CE and with pt≈0.968 it would have 1000× lower loss. This in turn increases the importance of correcting misclassified examples (whose loss is scaled down by at most 4× for pt≤.5 and γ=2).

In practice we use an α-balanced variant of the focal loss:
FL(pt)=−αt(1−pt)γlog(pt).(5)
View SourceRight-click on figure for MathML and additional features.We adopt this form in our experiments as it yields slightly improved accuracy over the non-α-balanced form. Finally, we note that the implementation of the loss layer combines the sigmoid operation for computing p with the loss computation, resulting in greater numerical stability.

While in our main experimental results we use the focal loss definition above, its precise form is not crucial. In Section 3.5 we consider other instantiations of the focal loss and demonstrate that these can be equally effective.

3.3 Class Imbalance and Model Initialization
Binary classification models are by default initialized to have equal probability of outputting either y=−1 or 1. Under such an initialization, in the presence of class imbalance, the loss due to the frequent class can dominate total loss and cause instability in early training. To counter this, we introduce the concept of a ‘prior’ for the value of p estimated by the model for the rare class (foreground) at the start of training. We denote the prior by π and set it so that the model’s estimated p for examples of the rare class is low, e.g., 0.01. We note that this is a change in model initialization (see Section 4.1) and not of the loss function. We found this to improve training stability for both the cross entropy and focal loss in the case of heavy class imbalance.”
From the above paper, summarize how focal loss for tackling the imbalanced dataset problem in deep learning and its advantages. Try to use your own words, do not copy the sentences directly from the above paper.