Dice Loss + Cross Entropy

username: Megh_Bhalerao
cooked: <p>Hello everyone,<br>
I don’t know if this is the right place to ask this but I’ll ask anyways.<br>
I am working on a multi class semantic segmentation problem, and I want to use a loss function which incorporates both dice loss &amp; cross entropy loss. How do I use this?<br>
I dont think a simple addition of dice score + cross entropy would make sense as the dice score is a small value between 0 &amp; 1, but the cross entropy value can also take very large values.<br>
So is there like some kind of normalization that should be performed on the CE loss to bring it in the same scale to that of the dice loss?<br>
Thanks in advance</p>

username: SoucheChapich
cooked: <p>Hello, I have exactly the same problem. Have you find something interesting ?</p>
<p>For the moment I am trying to normalize the CrossEntropyLoss by setting reduction parameter to True. The problem is as you said… CrossEntropy could take values bigger than 1.</p>
<p>I am actually trying with Loss = CE - log(dice_score) where dice_score is dice coefficient (opposed as the dice_loss where basically dice_loss = 1 - dice_score. I will wait for the results but some hints or help would be really helpful</p>

username: Megh_Bhalerao
cooked: <p>Hi, sorry for the late reply. I think what could be done for cross entropy loss is :<br>
Take the average of the negative log of the logits over 1 class and then average this value over all the classes. Doing this would give you a value in the similar scale to that of the dice loss with which you could simply add this CE value. Do you get what I mean? If not, let me know. But, when I try this method I keep getting NAN for my loss after few epochs (it is highly unpredictable when the loss would go to NAN). Did you get decent results by doing what you mentioned? i.e. taking the log of dice score?<br>
Please let me know</p>

username: SoucheChapich
cooked: <p>Hello, thank you for your reply.</p>
<p>I am not sure to have understood your point. CE works with the probabilities and then if you have a pixel with a very little probability to belong to a certain class (let´s say 1e-6), the value will be -log( p ) which in our case corresponds to ~14 which is far from the [0,1] range of the DL. I think that with nn.CrossEntropyLoss(), pytorch handle by default the normalization of the values (by taking the mean). But still at the end we don´t have the same range…</p>
<p>But as the values I obtain are close from [0,1] I simply add the two losses and the result is “good” in terms of mIoU. However, I truly believe that there is a “smarter” way to combine them and I am trying to finding that “smarter” way… Maybe something like DiceLoss + 0,7 * CE should be fine to “rescale” the CE</p>
<p>On the other hand, the results with the log of dice score doesn´t lead to great results, but I will experiment it again when my dataset will be bigger.</p>
<p>Finally, I don´t understand how you got NAN values from your loss, are you sure you reset the gradients properly ?</p>

username: Megh_Bhalerao
cooked: <p>Yes sorry I didn’t mean to say logits. I meant softmax probabilities. Yeah, I’ve seen the documentation of the nn.CrossEntropyLoss, I’ll try using that instead of writing my own functon as I was doing up untill now. Also, can you please tell me if the input to the nn.CrossEntropyLoss should be softmax probabilities or raw logits? I’m not very clear after reading the documentation.<br>
And regarding the gradients becoming NaN, I’m just doing torch.zero_grad()…is there anything else…I don’t know.<br>
Thanks for your reply</p>

username: andreaskoepf
cooked: <aside class="quote no-group" data-username="Megh_Bhalerao" data-post="5" data-topic="53194">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/megh_bhalerao/48/13547_2.png" class="avatar"> Megh_Bhalerao:</div>
<blockquote>
<p>can you please tell me if the input to the nn.CrossEntropyLoss should be softmax probabilities or raw logits?</p>
</blockquote>
</aside>
<p>You pass in ‘raw’ logits. The criterion itself passes them through a softmax normalization.<br>
See <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">nn.CrossEntropyLoss</a>: “This criterion combines <code>nn.LogSoftmax()</code> and <code>nn.NLLLoss()</code> in one single class.”</p>

username: SoucheChapich
cooked: <p>Yes as <a class="mention" href="/u/andreaskoepf">@andreaskoepf</a> said nn.CrossEntropyLoss combines both LogSoftmax and NLL that is why I did not use CE but NLL.</p>
<p>Are you doing something like optimizer.zero_grad() at the beginning of each epoch?</p>

username: Megh_Bhalerao
cooked: <p>Yeah I do that at the beginning of every iteration, not epoch. Anyways, I’ll use the inbuilt pytorch CE function and see whether I get NaN again. Thanks</p>

username: SoucheChapich
cooked: <p>Okay sure, let me know if it works.</p>

username: achaiah
cooked: <p>It’s a challenging problem but you can kind of reason about it this way - each loss prioritizes different features in your image. It is up to you to decide which features are most important and then weigh the losses such that the outcome is acceptable.  For example, dice loss puts more emphasis on imbalanced classes so if you weigh it more, your output will be more accurate/sensitive towards that goal. CE prioritizes the overall pixel-wise accuracy so some classes might suffer if they don’t have enough representation to influence CE.  Now when you add those two together you can play around with weighing the contributions of CE vs. Dice in your function such that the result is acceptable.</p>
<p>There are a myriad of different loss functions out there. <a href="https://github.com/achaiah/pywick/blob/master/pywick/losses.py" rel="nofollow noopener">Here</a> are the ones I’ve experimented so far. I’ve had good luck with <code>BCEDicePenalizeBorderLoss</code> but it’s BCE, not CE so not sure if it will work for you.</p>