Will PyTorch give different results from Tensorflow?

rostom007: <p>would like to reimplement a movie recommender system created in tensorflow with Pytorch.</p>
<p>My question is that does this modification affects the recommandation results?</p>

KFrank: <p>Hello Rostom!</p>
<aside class="quote no-group" data-username="rostom007" data-post="1" data-topic="79556" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/df788c/48.png" class="avatar"> rostom007:</div>
<blockquote>
<p>would like to reimplement a movie recommender system created in tensorflow with Pytorch.</p>
<p>My question is that does this modification affects the recommandation results?</p>
</blockquote>
</aside>
<p>Yes and no …</p>
<p>If you are careful, you should be able to reimplement a tensorflow<br>
network in pytorch and get the same* results.</p>
<p>*)  For some definition of “same” …</p>
<p>What could cause the results to differ?</p>
<p>Your two networks aren’t actually the same.  (You weren’t careful<br>
enough.)</p>
<p>The pytorch module / class / function that is supposed to match<br>
one in tensorflow doesn’t actually do the same thing.  This could<br>
be an error in pytorch or tensorflow, or could be a legitimate<br>
difference of opinion.  If this happens you might have to find a<br>
work-around, or reimplement tensorflow’s specific logic with your<br>
own pytorch code to get it to match.</p>
<p>Pseudo-random numbers – used for example to initialize weights<br>
and for things like <code>RandomSampler</code> – don’t match between<br>
tensorflow and pytorch (and, no, they don’t match).  In this case,<br>
the results of your two networks will differ, but they should agree<br>
<em>statistically</em>.  That is, if you rerun your training from scratch several<br>
times (using different random seeds each time), the <em>distribution</em><br>
of results you get from pytorch should match that from tensorflow.<br>
You could avoid this by not using tensorflow’s and pytorch’s<br>
built-in  random functions.  For example, you can initialize random<br>
weights and randomly sample data “by hand,” using your own same<br>
pseudo-random number generator for both.</p>
<p>The floating-point round-off differs in the two cases, because the<br>
order of (otherwise mathematically equivalent) operations differs.<br>
In this case your results should start out almost the same, only<br>
differing within floating-point round-off, and then drift apart as your<br>
training progresses.  Your results should still be statistically the<br>
same, but, in any individual run, could differ by a lot.  There is<br>
really no way to avoid this effect.</p>
<p>As a concrete example, I implemented the same simple digit<br>
classification network in both tensorflow and pytorch.  I had to<br>
dig around a little to figure out the exact logic that the two<br>
systems were using for their random weight initialization, and<br>
then tweak things a little to get them to match (statistically).<br>
After doing this, I was able to verify that I was getting <em>statistically</em><br>
equivalent results.  (I did not try to use my own pseudo-random<br>
number generator to get essentially identical results.)</p>
<p>Best.</p>
<p>K. Frank</p>

rostom007: <p>thnx for your reply.one more question , did u mean by (statistically) the same RMSE or MAE? this is exactly what i want to know.</p>