How to use Focal Loss for an imbalanced data for binary classification problem?

Mona_Jalal: <p>I have been searching in GitHub, Google, and PyTorch forum but it doesn’t seem there is a training for using PyTorch-based focal loss for an imbalanced dataset for binary classification. Further, there has been so many variation of the said loss. Is there any standardized version of this loss given its effectiveness and popularity inside the newer PyTorch library itself? If not, the experts in the field, which open-source implementation of the “focal loss” for “binary classification” in PyTorch do you suggest?</p>
<p>Further, if 14% of my dataset is the positive class, how do I assign these weights to the said focal loss when using it?</p>
<aside class="quote quote-modified" data-post="1" data-topic="128664">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b38774/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/implementing-focal-loss-for-a-binary-classification-problem/128664">Implementing Focal Loss for a binary classification problem</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    So I have been trying to implement Focal Loss recently (for binary classification), and have found some useful posts here and there, however, each solution differs a little from the other. Here, it’s less of an issue, rather a consultation. My question comes in two parts: 

I would like advice on the correctness of my -inspired by others- implementation (i.e., is it correct? if not, please advise.).
Is there any improvements to consider (e.g., efficiency or generalizability)?

Here is the snippe…
  </blockquote>
</aside>

<p>and</p>
<aside class="quote" data-post="1" data-topic="82417">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/umair_javaid/48/18886_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-get-alpha-values-of-focal-loss-for-multiclass-classification/82417">How to get alpha values of focal loss for multiclass classification?</a> 
  </div>
  <blockquote>
    I am sure the gamma value of focal loss is fixed(=2) but I am confused about how to get alpha values of focal loss for multiclass classification? Is the alpha value fixed or not? 
I do not understand the following line: 

It may be set by inverse class frequency
  </blockquote>
</aside>

<p>^ The questions posed above are left without an answer unfortunately.</p>
<p>For example, here’s another implementation, that I found on GitHub</p>
<pre><code class="lang-auto">def focal_loss_lgb_eval_error(y_pred, dtrain, alpha, gamma):
    """
    Adapation of the Focal Loss for lightgbm to be used as evaluation loss

    Parameters:
    -----------
    y_pred: numpy.ndarray
        array with the predictions
    dtrain: lightgbm.Dataset
    alpha, gamma: float
        See original paper https://arxiv.org/pdf/1708.02002.pdf
    """
    a,g = alpha, gamma
    y_true = dtrain.label
    p = 1/(1+np.exp(-y_pred))
    loss = -( a*y_true + (1-a)*(1-y_true) ) * (( 1 - ( y_true*p + (1-y_true)*(1-p)) )**g) * ( y_true*np.log(p)+(1-y_true)*np.log(1-p) )
    return 'focal_loss', np.mean(loss), False
</code></pre>
<aside class="onebox githubblob" data-onebox-src="https://github.com/jrzaurin/LightGBM-with-Focal-Loss/blob/master/examples/Lightgbm_with_Focal_Loss.ipynb">
  <header class="source">

      <a href="https://github.com/jrzaurin/LightGBM-with-Focal-Loss/blob/master/examples/Lightgbm_with_Focal_Loss.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/jrzaurin/LightGBM-with-Focal-Loss/blob/master/examples/Lightgbm_with_Focal_Loss.ipynb" target="_blank" rel="noopener nofollow ugc">jrzaurin/LightGBM-with-Focal-Loss/blob/master/examples/Lightgbm_with_Focal_Loss.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to code and use the Focal Loss with LightGBM\n",
    "\n",
    "The [Focal Loss](https://arxiv.org/pdf/1708.02002.pdf) for LightGBM can be coded as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pickle\n",
    "import lightgbm as lgb\n",
</code></pre>



  This file has been truncated. <a href="https://github.com/jrzaurin/LightGBM-with-Focal-Loss/blob/master/examples/Lightgbm_with_Focal_Loss.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Here’s another answer from PyTorch Forum:</p>
<pre><code class="lang-auto">BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')
pt = torch.exp(-BCE_loss) # prevents nans when probability 0
F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss
return focal_loss.mean()
</code></pre>
<aside class="quote quote-modified" data-post="5" data-topic="43327">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/diego/48/6989_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-this-a-correct-implementation-for-focal-loss-in-pytorch/43327/5">Is this a correct implementation for focal loss in pytorch?</a> 
  </div>
  <blockquote>
    Try this: 
BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')
pt = torch.exp(-BCE_loss) # prevents nans when probability 0
F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss
return focal_loss.mean()

Remember the alpha to address class imbalance and keep in mind that this will only work for binary classification. 
I think this is very similar to your implementation it just uses the BCE function which does the sigmoid and those .mul() for you and also prevents the n…
  </blockquote>
</aside>

<p>I think as a suggestion would be best that this “focal loss” be in the PyTorch itself.</p>

hamedB: <p>Hi <a class="mention" href="/u/mona_jalal">@Mona_Jalal</a></p>
<p>Thanks for sharing your experience. Kindly I have a question about alpha value in this loss. The common value is 0.25. Is this actually the weight of classes with lower amount of data (i.e., positive class)? In the Pytorch implementation of the weighted BCE loss (i.e., F.binary_cross_entropy_with_logits) the positive class (class with lower sample) has higher weights, close to one for example 0.7. And 1-0,7 as the weight for negative class (the class with the higher number of samples).</p>