Custom loss function error

username: edshkim98
cooked: <p>Hello everyone, I am currently trying to implement custom dice loss function for my semantic segmentation model.<br>
The problem is that I keep getting an error as below:<br>
<strong>’RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn’</strong><br>
I know that this error is caused due to using torch.argmax(pred,dim=1),<br>
as argmax function is not differentiable. However, without using argmax, I can not think of any other way as for my dice loss function I need to compare pred and target values as below:<br>
with torch.enable_grad():<br>
num= 2 * torch.sum(pred * target)<br>
den = torch.sum(pred + target)<br>
return 1 - torch.true_divide((num+ 1), (den+ 1))<br>
Could somebody please help me with this problem?<br>
Thank you!</p>
<p>My whole function looks as below:<br>
<strong>def custom_loss(output,labels):</strong><br>
**    preds = torch.nn.LogSoftmax(dim=1)(output)**<br>
**    preds = torch.argmax(preds,dim=1)**<br>
**    print(preds)**<br>
**    loss = dice_loss(preds,labels)**<br>
**    return loss**</p>
<p><strong>def dice_loss(pred,target):</strong><br>
**    with torch.enable_grad():**<br>
**        num= 2 * torch.sum(pred * target)**<br>
**        den = torch.sum(pred + target)**<br>
**        return 1 - torch.true_divide((num+ 1), (den+ 1))**</p>

username: tom
cooked: <p>Don’t use argmax but softmax and 1 component of the result.<br>
This will create a “soft dice”, which is actually differentiable (argmax does not have meaningful gradients).</p>

username: edshkim98
cooked: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="97818">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>softmax and 1 component</p>
</blockquote>
</aside>
<p>Hi, thank you for your reply!<br>
I am not sure I fully understand your answer.<br>
When I do nn.Softmax() then I get vectors including 2 different probabilities since it is binary segmentation model.<br>
Softmax function solely does not pick the highest probability index to compare with my target values, so surely I have to use functions like argmax after softmax function?</p>

username: tom
cooked: <p>I don’t think I can explain it much better than <a href="https://en.wikipedia.org/wiki/Softmax_function#Smooth_arg_max" rel="noopener nofollow ugc">wikipedia’s article on soft max</a>, but<br>
<code>torch.nn.functional.softmax(score, dim=1)[:, 1]</code> is actually a smooth approximation of <code>torch.argmax(score, dim=1)</code>. That is just what you need and the reason almost any classification works with argmax. We write a bit about that in section 7.2 <a href="https://pytorch.org/deep-learning-with-pytorch" rel="noopener nofollow ugc">of our book which can be freely downloaded at the moment</a> (7.2.3-7.2.5 in particular). We also use the dice loss in chapter 13 of the book.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: If you enclose your code in lines with triple backticks (```), you get source code formatting.</p>

username: edshkim98
cooked: <aside class="quote no-group" data-username="tom" data-post="4" data-topic="97818">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>torch.nn.functional.softmax(score, dim=1)[:, 1]</p>
</blockquote>
</aside>
<p>Thank you so much for your reply, I will have a go with reading your book!<br>
Just one more question, you said torch.nn.functional.softmax(score, dim=1)[:, 1] is same as torch.argmax(score,dim=1), but wouldn’t this always select the second index element no matter of probability of two elements?<br>
Here is an example when I tried your method:<br>
<strong>normal softmax:</strong>  tensor([[0.0850, 0.9150],<br>
[0.0598, 0.9402],<br>
[0.0765, 0.9235],<br>
…,<br>
[0.6879, 0.3121],<br>
[0.1176, 0.8824],<br>
[0.6816, 0.3184]], device=‘cuda:0’, grad_fn=)</p>
<p><strong>After:</strong> tensor([0.9150, 0.9402, 0.9235,  …, 0.3121, 0.8824, 0.3184], device=‘cuda:0’,<br>
grad_fn=)</p>
<p>The tensor after <strong>‘After:’</strong> is the result of using your recommended line(<strong>torch.nn.functional.softmax(score, dim=1)[:, 1]</strong>) and as you can see it always return the second index element’s probability not an index and no matter of the probabilities between the first and second elements.</p>

username: tom
cooked: <p>To form an intuition, make your values more extreme:</p>
<pre><code class="lang-python">scores = torch.randn(5, 2) * 10
print(scores.softmax(dim=1)[:, 1])
print(scores.argmax(dim=1))
</code></pre>
<p>this is what I mean by softmax being an approximation for argmax (of course this only works for 0 and 1 (two classes), in general softmax gives you an approximation of the 1-hot encoding of argmax).<br>
If you take away the <code>* 10</code> you have a similar effect (thresholding at 0.5 for the argmax), but the approximation is less close in general (which is good because gradients are better when it’s not so close to 0 or 1).</p>

username: edshkim98
cooked: <p>oh i see now why you said it’s an approximation.<br>
But in order to compare with my target values, I need values either 0 or 1 (I think). So, do you think whether rounding after  <strong>scores.softmax(dim=1)[:, 1]</strong> is a correct way of doing it?<br>
For example,</p>
<pre><code class="lang-auto">pred = scores.softmax(dim=1)[:, 1]
pred = torch.round(pred)
dice_loss(pred,target)

def dice_loss(pred,target):
    with torch.enable_grad():
        numerator = 2 * torch.sum(pred * target)
        denominator = torch.sum(pred + target)
        return 1 - torch.true_divide((numerator + 1), (denominator + 1))
</code></pre>

username: tom
cooked: <aside class="quote no-group" data-username="edshkim98" data-post="8" data-topic="97818">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edshkim98/48/29279_2.png" class="avatar"> edshkim98:</div>
<blockquote>
<p>But in order to compare with my target values, I need values either 0 or 1 (I think).</p>
</blockquote>
</aside>
<p>Well, that’s what they tell you in the definition of dice loss, but you don’t actually have to use it that way. If you want an intuition, think of the number as a probability of 1 in a Bernoulli distribution. Then the dice loss you get with probabilities is the one you would obtain (in the limit) if you would sample many copies and then take the dice loss over the lot of them (in one go, it’s not the expected dice loss but the dice loss of the expectation).</p>

username: edshkim98
cooked: <p>Thank you so much for the help!</p>

username: edshkim98
cooked: <p>Hi, just another question.<br>
Dice loss you have implemented on the book is for binary class.<br>
However, how can I construct dice loss for multiple classes?<br>
Thanks.</p>

username: tom
cooked: <p>Typically by taking 1-vs-all scores and then averaging them either weighted or unweighted.</p>