How to implement focal loss in pytorch?

Ben: <p>I implemented multi-class <a href="https://arxiv.org/abs/1708.02002" rel="nofollow noopener">Focal Loss</a> in pytorch. Bellow is the code. log_pred_prob_onehot is batched log_softmax in one_hot format, target is batched target in number(e.g. 0, 1, 2, 3).</p>
<pre><code>class FocalLoss(torch.nn.Module):
    def __init__(self, gamma=2):
        super().__init__()
        self.gamma = gamma

    def forward(self, log_pred_prob_onehot, target):
        pred_prob_oh = torch.exp(log_pred_prob_onehot)
        pt = Variable(pred_prob_oh.data.gather(1, target.data.view(-1, 1)), requires_grad=True)
        modulator = (1 - pt) ** self.gamma
        mce = modulator * (-torch.log(pt))

        return mce.mean()
</code></pre>
<p>However, when I tested it, it worked poorly. I read the Focal Loss paper a couple of times. It seems straight. Maybe I didn’ t understand it very well. I’d appreciate if anybody can correct me! Or if there is a workable implementation, please let me know! Thanks in advance!</p>
<p>Ben</p>

BowieHsu: <p><a href="https://github.com/kuangliu/pytorch-retinanet" rel="nofollow noopener">https://github.com/kuangliu/pytorch-retinanet</a> maybe this repo can solve your problem?</p>

Ben: <p>Thanks a lot <a class="mention" href="/u/bowiehsu">@BowieHsu</a>!</p>
<p>I also found this one is very good.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/DingKe/pytorch_workplace/blob/master/focalloss/loss.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/DingKe/pytorch_workplace/blob/master/focalloss/loss.py" target="_blank" rel="nofollow noopener">DingKe/pytorch_workplace/blob/master/focalloss/loss.py</a></h4>
<pre><code class="lang-py">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable


def one_hot(index, classes):
    size = index.size() + (classes,)
    view = index.size() + (1,)

    mask = torch.Tensor(*size).fill_(0)
    index = index.view(*view)
    ones = 1.

    if isinstance(index, Variable):
        ones = Variable(torch.Tensor(index.size()).fill_(1))
        mask = Variable(mask, volatile=index.volatile)

    return mask.scatter_(1, index, ones)

</code></pre>

  This file has been truncated. <a href="https://github.com/DingKe/pytorch_workplace/blob/master/focalloss/loss.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I ported this code to my program and it works.</p>

Ben: <p>I guess there is something wrong in the original code which breaks the computation graph and makes loss not decrease. I doubt it is this line:</p>
<pre><code>    pt = Variable(pred_prob_oh.data.gather(1, target.data.view(-1, 1)), requires_grad=True)
</code></pre>
<p>Is torch.gather support autograd? Is there anyway to implement this?<br>
Many thanks!</p>

BowieHsu: <p><a class="mention" href="/u/ben">@ben</a> Hi, ben,<br>
have you tried focal loss on SSD or FasteRcnn?<br>
How much mAP will be improved?</p>

Ben: <p><a class="mention" href="/u/bowiehsu">@BowieHsu</a> I used it in my own project, which has a multi-class, unbalanced data set. So far, it is not as good as it is mentioned in the paper. It is still very hard to train the data.</p>

trypag: <p>I also tried to use it in my own project, I found I had to reduce the lr by a factor of 10, leading to a better first iteration, but then due to the reduced lr, the precision over the epochs is barely improving. Maybe increasing lr after the first epoch could improve.</p>

xwy: <p>I tried it in my project, in FPN in faster rcnn, not as good as cross entropy though.</p>

Hengck: <p>i haven’t read the paper in deatils. But I thought the  the term (1-p)^gamma and p^gamma are for weighing only. They should not be back propagated during gradient descent. Maybe you need to detach() your variables?</p>

Hengck: <p>after some checking, the weighing terms (1-p)^gamma and p^gamma are back propagated as well. you can refer to:</p>
<p><a href="https://github.com/zimenglan-sysu-512/paper-note/blob/master/focal_loss.pdf" class="onebox" target="_blank" rel="nofollow noopener">https://github.com/zimenglan-sysu-512/paper-note/blob/master/focal_loss.pdf</a><br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/unsky/focal-loss" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/15643995?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/unsky/focal-loss" target="_blank" rel="nofollow noopener">unsky/focal-loss</a></h3>

<p>focal-loss - Focal loss for Dense Object Detection</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

Kento_Doi: <p>Hi Ben.<br>
Have you confirmed that training with gamma=0 is same to with cross entropy loss?<br>
I tried that in my implementation of focal loss. The result became very different <img src="//discuss.pytorch.org/images/emoji/apple/scream.png?v=5" title=":scream:" class="emoji" alt=":scream:"><img src="//discuss.pytorch.org/images/emoji/apple/scream.png?v=5" title=":scream:" class="emoji" alt=":scream:"></p>
<p>And I ask someone to answer <a href="https://discuss.pytorch.org/t/training-with-my-focal-loss-layer-doesnt-work/11293">my forum question</a>. I can’t identify the problem.</p>

Ben: <p>I found this one is pretty good, except some small grammar issues in python3. Enjoy!<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/marvis/pytorch-yolo2/blob/master/FocalLoss.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/marvis/pytorch-yolo2/blob/master/FocalLoss.py" target="_blank" rel="nofollow noopener">marvis/pytorch-yolo2/blob/master/FocalLoss.py</a></h4>
<pre><code class="lang-py">#!/usr/bin/env python
# -*- coding: utf-8 -*-
# --------------------------------------------------------
# Licensed under The MIT License [see LICENSE for details]
# Written by Chao CHEN (chaochancs@gmail.com)
# Created On: 2017-08-11
# --------------------------------------------------------
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

class FocalLoss(nn.Module):
    r"""
        This criterion is a implemenation of Focal Loss, which is proposed in 
        Focal Loss for Dense Object Detection.
            
            Loss(x, class) = - \alpha (1-softmax(x)[class])^gamma \log(softmax(x)[class])
    
        The losses are averaged across observations for each minibatch.
</code></pre>

  This file has been truncated. <a href="https://github.com/marvis/pytorch-yolo2/blob/master/FocalLoss.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

Kento_Doi: <p>Thank you for helping me, Ben!!</p>
<p>I completed implmentation of focal loss for semantic segmentation.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/doiken23/pytorch_toolbox/blob/master/focalloss2d.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/doiken23/pytorch_toolbox/blob/master/focalloss2d.py" target="_blank" rel="nofollow noopener">doiken23/pytorch_toolbox/blob/master/focalloss2d.py</a></h4>
<pre><code class="lang-py">####################################################
##### This is focal loss class for multi class #####
##### University of Tokyo Doi Kento            #####
####################################################

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
# I refered https://github.com/c0nn3r/RetinaNet/blob/master/focal_loss.py

class FocalLoss2d(nn.Module):

    def __init__(self, gamma=0, weight=None, size_average=True):
        super(FocalLoss2d, self).__init__()

        self.gamma = gamma
        self.weight = weight
        self.size_average = size_average

</code></pre>

  This file has been truncated. <a href="https://github.com/doiken23/pytorch_toolbox/blob/master/focalloss2d.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


edgarriba: <p>you can find now one_hot and focal loss implementations in torchgeometry:<br>
<a href="https://torchgeometry.readthedocs.io/en/latest/losses.html#torchgeometry.losses.one_hot" class="onebox" target="_blank" rel="nofollow noopener">https://torchgeometry.readthedocs.io/en/latest/losses.html#torchgeometry.losses.one_hot</a><br>
<a href="https://torchgeometry.readthedocs.io/en/latest/losses.html#torchgeometry.losses.FocalLoss" class="onebox" target="_blank" rel="nofollow noopener">https://torchgeometry.readthedocs.io/en/latest/losses.html#torchgeometry.losses.FocalLoss</a></p>

mattsim: <p>I’d like to add something to this, since it was leading me to an error.<br>
The implementation suggested by <a class="mention" href="/u/ben">@Ben</a> (<a href="https://github.com/marvis/pytorch-yolo2/blob/master/FocalLoss.py" rel="nofollow noopener">marvis/pytorch-yolo2/blob/master/FocalLoss.py</a>) beware specify alpha as a (C,1)-shaped tensor rather than a (C,).<br>
Otherwise, the implementation will be still working, but the loss will be computed as a dot product between the batch alpha values and the batch class probabilities, which makes conceptually no sense.</p>

edgarriba: <p>quick update: this can be found in kornia:<br>
<a href="https://kornia.readthedocs.io/en/latest/losses.html#kornia.losses.FocalLoss" class="onebox" target="_blank" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/losses.html#kornia.losses.FocalLoss</a></p>

edwardpwtsoi: <pre><code class="lang-python">class FocalLoss(torch.nn.Module):
    def __init__(self, gamma=2):
        super(FocalLoss, self).__init__()
        self.gamma = gamma

    def forward(self, inputs, targets):
        ps = F.softmax(inputs, dim=1)
        ne_ps = 1 - ps
        ws = torch.pow(ne_ps, self.gamma)
        return F.nll_loss(ps.pow(ws), targets)
</code></pre>
<p>Can anyone give me comment on my implementation?<br>
I was using it to solve classification problem with imbalance class<br>

but when I checked the TensorBoard logging, the loss is always the same
<br>
I changed to nll_loss and the loss is changing now, but I am still looking forward to having others comments <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Devraj_Mandal: <p><a class="mention" href="/u/edwardpwtsoi">@edwardpwtsoi</a>  I think if u are using NLL loss then u have to do logsoftmax.</p>

name_space: <p>How about the ignore_index apply to focal loss?</p>

PhilipMay: <p>Just FYI: This comment here:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/35882#issuecomment-721766641" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/35882#issuecomment-721766641" target="_blank" rel="noopener nofollow ugc">Focal loss implementation</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-02" data-time="15:13:07" data-timezone="UTC">03:13PM - 02 Apr 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-11-04" data-time="14:33:06" data-timezone="UTC">02:33PM - 04 Nov 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/b02202050" target="_blank" rel="noopener nofollow ugc">
          <img alt="b02202050" src="https://avatars2.githubusercontent.com/u/35265996?v=4" class="onebox-avatar-inline" width="20" height="20">
          b02202050
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
Implementation of Focal loss, which is first introduced in Tsung-Yi Lin, et al. "Focal Loss for Dense Object Detection," 2018..
Motivation
Focal...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: vision</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Sais that there is an implementation of focal loss in torchvision.</p>