Tensorflow SparseCategoricalCrossEntropy loss and Pytorch CrossEntropyLoss and adam optimizer

peter_vala: <p>Hi everyone,<br>
I’m trying to reproduce the training between tensorflow and pytorch. I came with a simple model using only one linear layer and the dataset that I’m using is the mnist hand digit. Before testing I assign the same weights in both models and then i calculate the loss for every single input. I noticed that some of the results are really close, but not actually the same. The cause i think that could be related to this small differences could be in the own losses implementation of both frameworks and be a round float error, could be this correct? I also test making a small training example using SGD optimizer and I noticed that the error becomes bigger along iterations. So could anyone show how both losses are implemented in both frameworks in order to test if the results become the same after?</p>
<p>The second part is about adam optimizer is the fact that i tried to implement by the pseudo code that i saw but the differences come to early and more significant than using SGD or SGD with momentum.So, why this happens?</p>

KFrank: <p>Hi Peter!</p>
<aside class="quote no-group" data-username="peter_vala" data-post="1" data-topic="153888" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_vala/48/45326_2.png" class="avatar"> peter_vala:</div>
<blockquote>
<p>some of the results are really close, but not actually the same. The cause i think that could be related to this small differences could be in the own losses implementation of both frameworks and be a round float error, could be this correct?</p>
</blockquote>
</aside>
<p>Such a discrepancy could well be caused by floating-point round-off error.<br>
Let’s assume that the tensorflow and pytorch implementations perform<br>
calculations that are mathematically identical.  It is, however, quite likely<br>
that some of the floating-point operations are performed in different, but<br>
mathematically equivalent orders, leading to differing round-off error.</p>
<p>I don’t know what tensorflow supports, but in pytorch it is easy to switch<br>
your calculations from single-precision (float) to double-precision (double)<br>
arithmetic.  If you can so this in tensorflow, you could rerun your loss<br>
computations in double precision.  You would still expect tensorflow and<br>
pytorch to produce results that differ by round-off error, but that difference<br>
should be reduced by several orders of magnitude.</p>
<aside class="quote no-group">
<blockquote>
<p>I also test making a small training example using SGD optimizer and I noticed that the error becomes bigger along iterations.</p>
</blockquote>
</aside>
<p>This makes sense.  Two things contribute to this:  As you run more training<br>
iterations, your round-off errors will accumulate.  But, furthermore, once<br>
your tensorflow and pytorch training runs become (a little bit) different<br>
because of one instance of (essentially trivial) round-off error, the paths<br>
in parameter space traced out by the training runs will continue to wander<br>
away from one another, even if you magically performed all subsequent<br>
computations with no round-off error.</p>
<aside class="quote no-group">
<blockquote>
<p>The second part is about adam optimizer is the fact that i tried to implement by the pseudo code that i saw but the differences come to early and more significant than using SGD or SGD with momentum.So, why this happens?</p>
</blockquote>
</aside>
<p>This also does not surprise me.  I can’t give a detailed explanation, but the<br>
Adam optimizer has the reputation of “jumping around more” and “being<br>
less stable.”  So once the round-off error causes the tensorflow and pytorch<br>
training runs to differ a little, the Adam optimizer might well cause the training<br>
paths to “wander away” from one another more rapidly.</p>
<p>Best.</p>
<p>K. Frank</p>

peter_vala: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>, first off all I hope everything is ok and thanks for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>Blockquote<br>
I don’t know what tensorflow supports, but in pytorch it is easy to switch<br>
your calculations from single-precision (float) to double-precision (double)<br>
arithmetic. If you can so this in tensorflow, you could rerun your loss<br>
computations in double precision. You would still expect tensorflow and<br>
pytorch to produce results that differ by round-off error, but that difference<br>
should be reduced by several orders of magnitude.</p>
</blockquote>
<p>Well my goal is to find and prove the causes of these differences, so i dont actually look for closer results, but the answer for those oscillations. Could you give me an code example of the loss implementation? I’ve tried to look in source code but I can’t understand some parts.</p>
<blockquote>
<p>Blockquote<br>
This also does not surprise me. I can’t give a detailed explanation, but the<br>
Adam optimizer has the reputation of “jumping around more” and “being<br>
less stable.” So once the round-off error causes the tensorflow and pytorch<br>
training runs to differ a little, the Adam optimizer might well cause the training<br>
paths to “wander away” from one another more rapidly.</p>
</blockquote>
<p>If I define the optimizer exactly the same wouldn’t that be same or would it be unstable as well?</p>
<p>Best regards,<br>
Peter Vala</p>

KFrank: <p>Hi Peter!</p>
<aside class="quote no-group" data-username="peter_vala" data-post="3" data-topic="153888" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_vala/48/45326_2.png" class="avatar"> peter_vala:</div>
<blockquote>
<p>Well my goal is to find and prove the causes of these differences</p>
</blockquote>
</aside>
<p>The cause of your differences is that even if tensorflow and pytorch are<br>
performing the same mathematical computations – and they may well<br>
be doing so – they are using mathematically equivalent, but numerically<br>
different orders of operations to do so, thereby incurring differing round-off<br>
errors.</p>
<aside class="quote no-group">
<blockquote>
<p>Could you give me an code example of the loss implementation?</p>
</blockquote>
</aside>
<p>If I were to give you a code example, you would just have a third version<br>
performing the computation with mathematically equivalent, but numerically<br>
different orders of operations.</p>
<p>There is (almost) no point in trying to chase down exactly what order of<br>
operations is being used or reproduce some specific round-off error.<br>
That’s just how floating-point arithmetic works (and you should rely on<br>
pytorch to be doing something fully reasonable).</p>
<aside class="quote no-group">
<blockquote>
<p>If I define the optimizer exactly the same wouldn’t that be same or would it be unstable as well?</p>
</blockquote>
</aside>
<p>If you were to define the optimizer <em>exactly</em> the same way – including<br>
ensuring that any sequences of mathematically equivalent  operations<br>
were performed in the same order so that they would be numerically<br>
equivalent as well – you would get the same result.</p>
<p>(But doing so would be a huge amount of work and not really of any value.)</p>
<p>Best.</p>
<p>K. Frank</p>