How to reproduce a Cross Entropy Loss without losing numerical accuracy?

tamerlan: <p>I get a discrepancy between the values of the losses obtained by the torch. CrossEntropyLoss and CustomeCrossEntropyLoss.</p>
<p><a href="https://github.com/pytorch/pytorch/issues/25990" rel="nofollow noopener">Issue on github</a></p>
<p>Code to Reproduce</p>
<pre><code class="lang-auto">import torch
import torch.nn.modules.loss as L  

class CustomCrossEntropyLoss(L._Loss):
    def __init__(self, reduction=True):
        super(CustomCrossEntropyLoss, self).__init__()
        self.reduction = reduction

     def forward(self, inp, target):
         input_target = inp.gather(1, target.view(-1, 1))
         input_max, _ = inp.max(dim=1, keepdim=True)
         output_exp = torch.exp(inp - input_max)
         output_softmax_sum = output_exp.sum(dim=1)
         output = -input_target + torch.log(output_softmax_sum).view(-1, 1) + input_max
         if self.reduction:
             output = output.mean()
         return output

 torch_ce = torch.nn.CrossEntropyLoss(reduction='none')
 custom_ce = CustomCrossEntropyLoss(reduction=False)
 batch_size = 128
 N_class = 90000
 logits = torch.randn((batch_size,N_class))
 targets = torch.randint(N_class, (batch_size,))
 print((torch_ce(logits, targets).view(-1) - custom_ce(logits,targets).view(-1)).mean())
</code></pre>
<p>I get a minimal non-zero discrepancy of the order of 10e-7, which then occurs in gradients and can affect the learning outcome.<br>
How to fix this problem?</p>

KFrank: <p>Hi Tamerlan!</p>
<aside class="quote no-group quote-modified" data-username="tamerlan" data-post="1" data-topic="55869" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/bbe5ce/48.png" class="avatar"> tamerlan:</div>
<blockquote>
<p>I get a discrepancy between the values of the losses obtained by the torch. CrossEntropyLoss and CustomeCrossEntropyLoss.</p>
<p>Code to Reproduce</p>
<pre><code class="lang-auto">...
 logits = torch.randn((batch_size,N_class))
 targets = torch.randint(N_class, (batch_size,))
 print((torch_ce(logits, targets).view(-1) - custom_ce(logits,targets).view(-1)).mean())
</code></pre>
<p>I get a minimal non-zero discrepancy of the order of 10e-7, which then occurs in gradients and can affect the learning outcome.<br>
How to fix this problem?</p>
</blockquote>
</aside>
<p>I haven’t traced through your code in detail, nor looked at<br>
your github issue, but the explanation is likely the following:</p>
<p>Your <code>logits</code> tensor is, by default (unless you change the<br>
global default), of <code>dtype = torch.float32</code>.  32-bit floating<br>
point numbers have about 7 decimal digits of precision.</p>
<p>Even though (presumably – I haven’t checked your code)<br>
<code>torch_ce</code> and <code>custom_ce</code> should be <em>mathematically</em><br>
equivalent, they likely do not perform their calculations in<br>
exactly the same order, so floating-point round-off error<br>
can and will lead to slight differences.</p>
<p>Your <code>logits</code> (<code>randn</code>) are of order one, so you should<br>
reasonably expect the round-off-error differences you<br>
see of order 10e-7.</p>
<p>There isn’t any “fix” for this – this is how floating-point<br>
arithmetic works.</p>
<p>If you were to go through <code>torch_ce</code> and modify your<br>
<code>custom_ce</code> so that the two perform their calculations<br>
in exactly the same order, you ought to be able to get<br>
them to produce exactly the same result.  But there<br>
would be no point in that – your result would still differ<br>
from the “true” mathematical result by a round-off error<br>
on the order of 10e-7.</p>
<p>If for some reason you need greater precision, you could<br>
use 64-bit floating point arithmetic.  Try adding<br>
<code>dtype = torch.float64</code> to your <code>torch.randn()</code> call.<br>
If my explanation above is correct, you should now see<br>
differences of the order of 10e-15.</p>
<p>Good luck.</p>
<p>K. Frank</p>