Focal loss in pytorch

username: ni_tempe
cooked: <p>I have binary NLP classification problem and my data is very biased. Class 1 represents only 2% of data. For training I am oversampling from class 1 and for training my class distribution is 55%-45%. I have built a CNN. My last few layers and loss function as below</p>
<pre><code>self.batch_norm2 = nn.BatchNorm1d(num_filters)
        
self.fc2 = nn.Linear(np.sum(num_filters), fc2_neurons)
        
self.batch_norm3 = nn.BatchNorm1d(fc2_neurons)
        
self.fc3 = nn.Linear(fc2_neurons, 1)#changing on 6March - BCE with logits loss
</code></pre>
<p>Loss:</p>
<pre><code>BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_examples[0]/class_examples[1])
</code></pre>
<p>In my evaluation function I am calling that loss as follows</p>
<pre><code>loss=BCE_With_LogitsLoss(torch.squeeze(probs), labels.float())
</code></pre>
<p>I was suggested to use focal loss over <a href="https://stackoverflow.com/questions/71462326/pytorch-bcewithlogitsloss-calculating-pos-weight/71494404#71494404" rel="noopener nofollow ugc">here</a>.</p>
<p>Please consider using <a href="https://stackoverflow.com/a/52161194/1714410" rel="noopener nofollow ugc">Focal loss</a>:<br>
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár <a href="https://arxiv.org/abs/1708.02002" rel="noopener nofollow ugc">Focal Loss for Dense Object Detection</a> (ICCV 2017).</p>
<p>Is there any pytorch implementation of the same? I found few but now sure which are correct.<br>
one example - <a href="https://stackoverflow.com/questions/71300607/using-focal-loss-for-imbalanced-dataset-in-pytorch" rel="noopener nofollow ugc">https://stackoverflow.com/questions/71300607/using-focal-loss-for-imbalanced-dataset-in-pytorch</a><br>
another example - <a href="https://stackoverflow.com/questions/66178979/focal-loss-implementation" rel="noopener nofollow ugc">https://stackoverflow.com/questions/66178979/focal-loss-implementation</a></p>
<p>My questions:</p>
<ol>
<li>is there any specific implementation that I should use?</li>
<li>Should I use focal loss even though I am oversampling?</li>
<li>How to modify my code to use the correct implementation</li>
<li>do we need to use <code>pos_weight</code> along with it?</li>
</ol>

username: ptrblck
cooked: <ol>
<li>I would probably use Kornia’s implementation from <a href="https://kornia.readthedocs.io/en/latest/_modules/kornia/losses/focal.html">here</a> (CC <a class="mention" href="/u/edgarriba">@edgarriba</a>)</li>
<li>I’m unsure what your current oversampling strategy is as each sample contains multiple targets (each pixel is associated with a class)</li>
<li>Unsure, how to understand the question, but you should be able to use the linked loss implementation directly in your code after installing <code>kornia</code>
</li>
<li>I don’t think focal loss uses <code>pos_weights</code>
</li>
</ol>

username: ni_tempe
cooked: <p>appreciate the answer</p>
<ol>
<li>Thanks for the link.</li>
<li>trying to clarify -  This is a NLP problem and I dont have any images as input. Lets say my master data has 100,000 examples of class 0 and 20,000 class 1 then my training data has 10,000 class 0 and 10,000 class 1. I oversampled from class 1. Each example is some text and associate y could be either 0 or 1. My test data still has 98%-2% distribution. <em>In such case of oversampling should I still us e focal loss?</em>
</li>
</ol>

username: ptrblck
cooked: <p>I’m not familiar with using focal loss for NLP use cases and don’t know if it would work (it’s definitely worth an experiment so let us know how it went <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> ).</p>

username: ni_tempe
cooked: <p>I will definitely update with my results.</p>
<p>quick question - why 2 lines related to <code>BCE_With_LogitsLos</code>s loss work but similar 2 lines with <a href="https://kornia.readthedocs.io/en/latest/losses.html?highlight=losses" rel="noopener nofollow ugc">focal</a> loss fail?</p>
<pre><code class="lang-auto">BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_examples[0]/class_examples[1])
loss=BCE_With_LogitsLoss(torch.squeeze(probs), labels.float())
</code></pre>
<p><code>Focal loss</code> - below lines give error. is it because for focal loss <code>input</code> and <code>target</code> are mandatory? why they are not mandatory for <code>BCE_With_LogitsLoss</code>??</p>
<pre><code class="lang-auto">focal_loss=kornia.losses.binary_focal_loss_with_logits()
loss=focal_loss(torch.squeeze(probs), labels.float())
</code></pre>

username: ptrblck
cooked: <p>I don’t know where <code>binary_focal_loss_with_logits</code> is coming from, as it doesn’t seem to be defined in <code>kornia</code>.<br>
I also think that both losses expect to get a target tensor as well as an input and don’t know which error you are seeing.</p>

username: ni_tempe
cooked: <p>BCE_With_LogitsLoss is from <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">pytorch</a> directly as shown in the below line<br>
<code>BCE_With_LogitsLoss=nn.BCEWithLogitsLoss(pos_weight=class_examples[0]/class_examples[1])</code></p>
<p>I will post error with focal loss soon</p>

username: ni_tempe
cooked: <p>so far not seeing huge improvement as compared to <code>nn.BCEWithLogitsLoss</code></p>