Verify the implementation of loss function

username: richardatpytorch
cooked: <p>I am trying to implement the Generalized Dice Loss function by myself. When I tested the output of my implementation with <a href="https://docs.monai.io/en/stable/_modules/monai/losses/dice.html#GeneralizedDiceLoss" rel="noopener nofollow ugc">monai</a>, I got the same result. However, the training losses of these two functions were different when I used them in a training pipeline. I do not know what is the reason for this difference, and I am also not sure about the way I compare the two loss functions. Below is the code I used to test the implementation:</p>
<pre><code class="lang-auto">import torch
from loss_factory import LossFunctionType, LossFunctionFactory
from monai import losses


loss_fn = LossFunctionFactory.construct_loss_function(LossFunctionType.GENERALIZED_DICE)
monai_loss_fn = losses.dice.GeneralizedDiceLoss(to_onehot_y=True, softmax=True)
num_classes = 2

inputs = torch.randn((1, num_classes, 3, 4,), requires_grad=True)
targets = torch.empty((1, 1, 3, 4), dtype=torch.long).random_(num_classes)
loss = loss_fn(inputs, targets)
print(loss)

inputs_ = inputs.detach().clone().requires_grad_()
monai_loss = monai_loss_fn(inputs_, targets)
print(monai_loss)
</code></pre>

username: KFrank
cooked: <p>Hi Richard!</p>
<aside class="quote no-group quote-modified" data-username="richardatpytorch" data-post="1" data-topic="189823" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richardatpytorch/48/59675_2.png" class="avatar"> richardatpytorch:</div>
<blockquote>
<p>I am also not sure about the way I compare the two loss functions.<br>
…</p>
<pre><code class="lang-auto">inputs = torch.randn((1, num_classes, 3, 4,), requires_grad=True)
targets = torch.empty((1, 1, 3, 4), dtype=torch.long).random_(num_classes)
loss = loss_fn(inputs, targets)
print(loss)

inputs_ = inputs.detach().clone().requires_grad_()
monai_loss = monai_loss_fn(inputs_, targets)
print(monai_loss)
</code></pre>
</blockquote>
</aside>
<p>If I understand you correctly, you are saying that the two computed loss values<br>
are equal (up to potential round-off error), but that training with the two loss<br>
values differs.</p>
<p>This suggests that you should also test the gradients of the two losses:</p>
<pre data-code-wrap="python"><code class="lang-python">loss.backward()         # computes the gradient of your implementation
print (inputs.grad)     # print your gradient

monai_loss.backward()   # computes the gradient of the monai implementation
print (inputs_.grad)    # print the monai gradient

print (torch.allclose (inputs.grad, inputs_.grad, atol = 1.e-4)   # check equality to some appropriate tolerance
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: richardatpytorch
cooked: <p>I have followed your suggestion and the result still shows that these 2 functions produce the same</p>
<pre><code class="lang-auto">tensor(0.5856, grad_fn=&lt;MeanBackward0&gt;)
tensor(0.5856, grad_fn=&lt;MeanBackward0&gt;)
tensor([[[[ 0.0097,  0.0145,  0.0016,  0.0126],
          [ 0.0150, -0.0162,  0.0149, -0.0120],
          [-0.0283,  0.0107,  0.0146, -0.0294]],

         [[-0.0097, -0.0145, -0.0016, -0.0126],
          [-0.0150,  0.0162, -0.0149,  0.0120],
          [ 0.0283, -0.0107, -0.0146,  0.0294]]]])
tensor([[[[ 0.0097,  0.0144,  0.0016,  0.0126],
          [ 0.0150, -0.0162,  0.0149, -0.0120],
          [-0.0283,  0.0107,  0.0145, -0.0294]],

         [[-0.0097, -0.0144, -0.0016, -0.0126],
          [-0.0150,  0.0162, -0.0149,  0.0120],
          [ 0.0283, -0.0107, -0.0145,  0.0294]]]])
True
</code></pre>

username: KFrank
cooked: <p>Hi Richard!</p>
<aside class="quote no-group" data-username="richardatpytorch" data-post="3" data-topic="189823" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richardatpytorch/48/59675_2.png" class="avatar"> richardatpytorch:</div>
<blockquote>
<p>the result still shows that these 2 functions produce the same</p>
</blockquote>
</aside>
<p>Okay, that looks good.</p>
<p>Here are three more debugging suggestions:</p>
<p>First, make sure any randomness in your training pipeline – shuffling in your<br>
dataloader, dropouts, etc. – is the same in the two training runs you are<br>
comparing.  If it’s not, you could easily get large differences in your results<br>
that have nothing to do with the two version of your loss function.</p>
<p>Second, when you train, small differences due to expected round-off error can<br>
accumulate over many optimization cycles.  The two training runs can wander<br>
off in two (equally-good) directions, leading to results that differ by a lot.</p>
<p>Try optimizing with plain-vanilla <code>SGD</code> with no momentum and a small learning<br>
rate.  Do your results agree (within a smallish multiple of round-off error) after<br>
a single optimization step?  Do they continue to agree reasonably well after a<br>
handful of optimization steps?  If your two training runs diverge from one another<br>
only rather slowly, things are probably fine, even if the two runs end up with<br>
distinctly different results.</p>
<p>Third, it could be that the two versions of your loss function agree for the<br>
majority of inputs, but there are some edge cases (for example, maybe the<br>
two versions use a different value of epsilon to protect against division by<br>
very small values or zero in the Dice-coefficient computation) where they<br>
give significantly different results.  You could try running your consistency<br>
check (including the gradient) on a large number of random inputs.  You might<br>
also consider salting your random inputs with possible edge-case examples.</p>
<p>Good luck!</p>
<p>K. Frank</p>