Using Focal loss for multilabel classification problem

gmanoj: <p>I am doing a multi label classification problem. Output is a 199 dimension vector of 0’s and 1’s . I am using nn.BCELoss() and output of the model are the predicted probabilities.<br>
Now I have two queries.</p>
<ol>
<li>Can focal loss be used with multilabel classification problem. As per research paper of focal loss , cross entropy loss was used with focal loss which I can’t use here. If yes, I would be grateful if someone can provide reference documentation on how to use this loss function for multilabel case.</li>
<li>I have two types of imbalances. Suppose my output is a 5 dim vector = [0,1,0,0,0]. One imbalance is that for each example there are mostly zeroes and only few bits are turned on. For me 1’s are very important. Another type of imbalance is that across different examples, there are few majority classes (like there is 1 at always 2nd index indicating class 2 occurs in majority examples) and some minority classes. What will be the correct way to deal with these two type of imbalances and how to handle this with inbuilt pytorch functionality?</li>
</ol>

Soumya_Kundu: <p>For your first question, you can refer to this: <a href="https://www.kaggle.com/code/thedrcat/focal-multilabel-loss-in-pytorch-explained" class="inline-onebox" rel="noopener nofollow ugc">Focal Multilabel Loss in Pytorch Explained | Kaggle</a></p>
<p>For second query; In the first scenario, the normal implementation would work well. In the second scenario, maybe look into class weighting or some type of custom sampling? Maybe of interest: <a href="https://www.sciencedirect.com/science/article/pii/S003132032100474X" rel="noopener nofollow ugc">https://www.sciencedirect.com/science/article/pii/S003132032100474X</a></p>

gmanoj: <p>For first question, isn’t there a direct implementation of focal loss for multilabel case in pytorch? There is one documentation I found but unable to understand if this can be used directly for my use-case.<br>
<a href="https://pytorch.org/vision/main/generated/torchvision.ops.sigmoid_focal_loss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.ops.sigmoid_focal_loss.html</a></p>

Soumya_Kundu: <p>Could maybe use the following as a skeleton:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class FocalLoss(nn.Module):
    def __init__(self, alpha=1, gamma=2, logits=False, reduce=True):
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = gamma
        self.logits = logits
        self.reduce = reduce

    def forward(self, inputs, targets):
        if self.logits:
            BCE_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none')
        else:
            BCE_loss = F.binary_cross_entropy(inputs, targets, reduction='none')
        pt = torch.exp(-BCE_loss)
        F_loss = self.alpha * (1-pt)**self.gamma * BCE_loss

        if self.reduce:
            return torch.mean(F_loss)
        else:
            return F_loss
</code></pre>
<p>You could also potentially look into Asymmetric Loss: <a href="https://openaccess.thecvf.com/content/ICCV2021/papers/Ridnik_Asymmetric_Loss_for_Multi-Label_Classification_ICCV_2021_paper.pdf" rel="noopener nofollow ugc">https://openaccess.thecvf.com/content/ICCV2021/papers/Ridnik_Asymmetric_Loss_for_Multi-Label_Classification_ICCV_2021_paper.pdf</a></p>

gmanoj: <p>Kaggle link shared worked fine for me, thanks</p>