Focal loss for imbalanced multi class classification in Pytorch

VikasRajashekar: <p>I want an example code for Focal loss in PyTorch for a model with three class prediction. My model outputs 3 probabilities.</p>
<p>Sentiment_LSTM(<br>
(embedding): Embedding(19612, 400)<br>
(lstm): LSTM(400, 512, num_layers=2, batch_first=True, dropout=0.5)<br>
(dropout): Dropout(p=0.5, inplace=False)<br>
(fc): Linear(in_features=512, out_features=3, bias=True)<br>
(sig): Sigmoid() )</p>
<p>My class distribution is highly imbalanced. So I want to try focal loss so that the minor class accuracy is improved.</p>
<p>I currently used loss function defined in <a href="https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65938" rel="nofollow noopener">https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65938</a> But it didn’t help.</p>
<p>The original paper(<a href="https://arxiv.org/abs/1708.02002" rel="nofollow noopener">https://arxiv.org/abs/1708.02002</a>) only consider binary classification. How do I extend it to the multi-class scenario?</p>

tom: <p>I don’t think you would want sigmoid for multi-class (I’m assuming you mean multi-class rather than multi-label and already train with (unfocused - ha!) cross entropy loss).<br>
If your regular cross entropy loss is “ce_loss”, you can just define alpha and gamma and do as in the linked function</p>
<pre><code class="lang-auto">ce_loss = torch.nn.functional.cross_entropy(outputs, targets, reduction='none') # important to add reduction='none' to keep per-batch-item loss
pt = torch.exp(-ce_loss)
focal_loss = (alpha * (1-pt)**gamma * ce_loss).mean() # mean over the batch
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

VikasRajashekar: <p>Here is my network def: I am not usinf the sigmoid layer as cross entropy takes care of it. so I pass the raw logits to the loss function</p>
<pre><code class="lang-auto">import torch.nn as nn

class Sentiment_LSTM(nn.Module):
    """
    We are training the embedded layers along with LSTM for the sentiment analysis
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        """
        Settin up the parameters.
        """
        super(Sentiment_LSTM, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        
        # embedding layer and LSTM layers 
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, 
                            dropout=drop_prob, batch_first=True)
        
        # dropout layer to avoida over fitting
        self.dropout = nn.Dropout(0.5)
        
        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        self.sig = nn.Sigmoid()
        

    def forward(self, x):
        """
        Perform a forward pass

        """
        batch_size = x.size(0)

        x = x.long()
        embeds = self.embedding(x)

        lstm_out, hidden = self.lstm(embeds)

    
        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)


        out = self.dropout(lstm_out)
        out = self.fc(out)

        # sigmoid function
        sig_out = out
        
        # reshape to be batch_size first
        sig_out = sig_out.view(batch_size, -1,3)
        #print("sig_out",sig_out.shape)
        sig_out = sig_out[:, -1,:] # get last batch of labels
        
        # return last sigmoid output and hidden state
        return sig_out
    
    
    def init_hidden(self, batch_size):
        #initilizing hidden layers
        weight = next(self.parameters()).data
        
        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())
        
        return hidden
</code></pre>
<p>My loss function:</p>
<pre><code class="lang-auto">class FocalLoss(nn.Module):
    def __init__(self, alpha=1, gamma=2, logits=False, reduce=True):
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = gamma
        self.logits = logits
        self.reduce = reduce

    def forward(self, inputs, targets):nn.CrossEntropyLoss()
    
        BCE_loss = nn.CrossEntropyLoss()(inputs, targets, reduce=False)

        pt = torch.exp(-BCE_loss)
        F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss

        if self.reduce:
            return torch.mean(F_loss)
        else:
            return F_loss
</code></pre>
<p>My output has size of 3: It has to predict if the sentiment is positive , negative or neutral.<br>
I data is imblanced. Neutral is around 7000, positve around 250 and negative around 800. Is my understanding and the implementation makes sense?</p>

tom: <aside class="quote no-group" data-username="VikasRajashekar" data-post="3" data-topic="61289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vikasrajashekar/48/19580_2.png" class="avatar"> VikasRajashekar:</div>
<blockquote>
<p>BCE_loss = nn.CrossEntropyLoss()(inputs, targets, reduce=False)</p>
</blockquote>
</aside>
<p>This doesn’t look right.</p>
<ul>
<li>You probably just want the functional version (see above) and pass <code>reduction='none'</code> to be modern.</li>
<li>I’d not call it BCE_loss.</li>
</ul>
<p>My impression is that focal loss may help, but there are quite a few ways to do this, the most simple one is balanced sampling during training and a recent one is the weighted loss function from <a href="https://arxiv.org/abs/1901.05555" class="inline-onebox" rel="noopener nofollow ugc">[1901.05555] Class-Balanced Loss Based on Effective Number of Samples</a> .</p>
<p>Best regards</p>
<p>Thomas</p>

Umair_Javaid: <p>Can you answer this aswell?</p><aside class="quote" data-post="1" data-topic="82417">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/umair_javaid/48/18886_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-get-alpha-values-of-focal-loss-for-multiclass-classification/82417">How to get alpha values of focal loss for multiclass classification?</a> 
  </div>
  <blockquote>
    I am sure the gamma value of focal loss is fixed(=2) but I am confused about how to get alpha values of focal loss for multiclass classification? Is the alpha value fixed or not? 
I do not understand the following line: 

It may be set by inverse class frequency
  </blockquote>
</aside>


Umair_Javaid: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="61289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p><code>focal_loss = (alpha * (1-pt)**gamma * ce_loss).mean()</code></p>
</blockquote>
</aside>
<p>what’s the value of alpha here?</p>

tom: <p><code>alpha</code> is an additional weighting factor between classes. In the paper linked above it is introduced in eq (5).</p>
<p>Best regards</p>
<p>Thomas</p>

shakeel608: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="61289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<pre><code class="lang-auto">pt = torch.exp(-ce_loss)
</code></pre>
</blockquote>
</aside>
<p>Why we need use the below line in focal loss, as per paper,  pt = p if y==1, otherwise 1-p</p>
<pre><code class="lang-auto">torch.exp(-ce_loss)

</code></pre>
<p>Why torch.exp ?? What does it reflect here</p>

tom: <p>The <code>ce_loss</code> is a negative log likelihood and so <code>torch.exp(-ce_loss)</code> is the likelihood (i.e. between 0 and 1 etc.).</p>

shakeel608: <p>I got it ,<br>
but why do we need torch.exp here<br>
Could you please clarify it with a simple example</p>

tom: <p>I’m not sure I understand? <code>torch.exp</code> of a <strong>log</strong> likelihood gives you the likelihood because <code>exp</code> is the inverse operation of <code>log</code>.</p>

XavierMFC: <p><strong>alpha</strong> is really a hard hyper-parameters。。。</p>

Xinchengzelin: <p>alpha can’t be the balanced factor as the paper,  it’s just the scaled factor, right?</p>

MaxWolf-01: <p>Would it be correct to use alpha already in the <code>cross_entropy</code> calculation as weight like this?</p>
<pre><code class="lang-auto">ce_loss = torch.nn.functional.cross_entropy(outputs, targets, reduction='none', weight=alpha) 
pt = torch.exp(-ce_loss)
focal_loss = ((1-pt)**gamma * ce_loss).mean()
</code></pre>

Shix0: <p><a class="mention" href="/u/maxwolf-01">@MaxWolf-01</a> I was wondering about the same idea. I’ve tried that, and I am getting unstable loss values (keep fluctuating between extremes). Without using class weights (i.e., weight=None), the loss values are stable, but Focal loss overfits in comparison to <strong>nn.CrossEntropyLoss</strong> with class weights in that case.</p>
<p><a class="mention" href="/u/tom">@tom</a>  Any ideas here?</p>
<p>Here is the implementation:</p>
<blockquote>
<p>class FocalLoss(torch.nn.Module):<br>
“”"Implementation of the Focal loss function</p>
<pre><code>    Args:
        weight: class weight vector to be used in case of class imbalance
        gamma: hyper-parameter for the focal loss scaling.
"""
def __init__(self, weight=None, gamma=2):
    super(FocalLoss, self).__init__()
    self.gamma = gamma
    self.weight = weight #weight parameter will act as the alpha parameter to balance class weights

def forward(self, outputs, targets):
    ce_loss = torch.nn.functional.cross_entropy(outputs, targets, reduction='none', weight=self.weight) 
    pt = torch.exp(-ce_loss)
    focal_loss = ((1-pt)**self.gamma * ce_loss).mean() # mean over the batch
    return focal_loss
</code></pre>
</blockquote>