Negative loss after transfering weights from Tensorflow to Pytorch

username: peter_vala
cooked: <p>Hey everyone,<br>
I’m working in a translation between both frameworks and I already manage the translation of the weights on both of them. However i found one problem: Although I have the same accuracy on predictions, the loss is different but not just different it is negative on Pytorch side while is positive in Tensorflow side. Does anyone can help me figure it out what’s wrong?</p>

username: tom
cooked: <p>If this is a loss that should not have negative values, quite likely something is up with the inputs to the loss.</p>
<p>There are many subtle differences between TF and PyTorch. I usually try to start with a fixed input and then successively make sure both computations have every intermediate result exactly the same. (Or expected change, like the ordering of dimensions, which can be different between the two, too.) This is a bit tricky when randomness is involved but usually gives good results.</p>
<p>Best regards</p>
<p>Thomas</p>

username: peter_vala
cooked: <p><a class="mention" href="/u/tom">@tom</a> thanks for the reply.<br>
So at first place the input I’m using the same images for both frameworks and the only thing I do is change the shape of the image to be suitable for each one.<br>
I had some problems at the beginning for the conv2d layers and I have same intermediate results reallly close to be the same but they change in the order of ~= 1e-5 ,1e-6, values that I give the cause of round calculations or could I be mistaken?</p>

username: tom
cooked: <aside class="quote no-group" data-username="peter_vala" data-post="3" data-topic="146740">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_vala/48/45326_2.png" class="avatar"> peter_vala:</div>
<blockquote>
<p>I had some problems at the beginning for the conv2d layers and I have same intermediate results reallly close to be the same but they change in the order of ~= 1e-5 ,1e-6, values that I give the cause of round calculations or could I be mistaken?</p>
</blockquote>
</aside>
<p>Yes, so I was missing a “up to numerical accuracy” in my original reply.<br>
But so the output is similarly close then and just the loss wrecks havoc or do the intermediates stop being really close?</p>
<p>Best regards</p>
<p>Thomas</p>

username: peter_vala
cooked: <p>Thanks again for the reply <a class="mention" href="/u/tom">@tom</a>. I think the best way to explain my problem is giving a little example of what’s going on.<br>
So I made a small test with only 1 conv2d layer and 1 linear layer. The input_shape of the models is 1 channel image 5x5 resized from mnist hand digit dataset with the respective targets. Every layer is simple with ‘valid’ pad for tensorflow (1,1) stride and (3x3) kernel with only 2 filters output for conv2d and no activation functions just the softmax in the linear layer. The configuration is not important, at least every other configuration that i gave to the layers didnt drop the accuracy as the only problem being the loss of predictions.</p>
<p>The ouput after softmax from Tensorflow is the follow:</p>
<pre><code class="lang-auto">[1.1379382e-02 1.9076133e-05 4.2098956e-03 4.5368117e-03 5.9396513e-03
 7.0751053e-03 8.0966354e-05 9.4762051e-01 1.0554336e-02 8.5842358e-03], shape=(10,), dtype=float32)
</code></pre>
<p>As the Pytorch output is:</p>
<pre><code class="lang-auto">tensor([[1.1379386298e-02, 1.9076149329e-05, 4.2099035345e-03, 4.5368201099e-03,
         5.9396508150e-03, 7.0751085877e-03, 8.0966427049e-05, 9.4762051105e-01,
         1.0554354638e-02, 8.5842274129e-03]])
</code></pre>
<p>After the predictions overall dataset this are the results:</p>
<ol>
<li>Tensorflow: 313/313 - 1s - loss: 1.2292 - accuracy: 0.5802 - 638ms/epoch - 2ms/step</li>
<li>Pytorch: Test set: Average loss: -0.4224, Accuracy: 5802/10000 (58%)</li>
</ol>
<p>Just in case you want to give a look on intermediate step this are the results after conv2d:</p>
<ol>
<li>Tensorflow</li>
</ol>
<pre><code class="lang-auto">tf.Tensor(
[[[-2.04286    -0.03277442]
  [-1.8308406  -0.661551  ]
  [-1.6251079   0.14818816]]

 [[ 0.37748477 -0.05223539]
  [-0.0944322   3.0911071 ]
  [ 0.30482826 -0.00778666]]

 [[-0.10668626 -0.02989659]
  [-0.69553125  1.2320076 ]
  [-0.20329499 -0.3670036 ]]], shape=(3, 3, 2), dtype=float32)
</code></pre>
<ol start="2">
<li>Pytorch</li>
</ol>
<pre><code class="lang-auto">tensor([[[[-2.0428609848, -1.8308403492, -1.6251083612],
          [ 0.3774845302, -0.0944317281,  0.3048284352],
          [-0.1066871583, -0.6955307126, -0.2032952011]],

         [[-0.0327757746, -0.6615512371,  0.1481878012],
          [-0.0522354990,  3.0911066532, -0.0077868253],
          [-0.0298968107,  1.2320076227, -0.3670037389]]]])
</code></pre>

username: tom
cooked: <p>So the output is close enough to consider it the same.<br>
But are you sure you are using the same loss?<br>
In particular, the popular <code>CrossEntropyLoss</code> in PyTorch wants logits as inputs, not probabilities.</p>
<p>Best regards</p>
<p>Thomas</p>

username: peter_vala
cooked: <p>Well, I was using nll_loss in Pytorch i changed to CrossEntropyLoss function and in Keras i use sparse_categorical_crossentropy.<br>
The results are now positive, but they are not the same as the result to Pytorch i got 1.9976  and Keras  1.2292 is this normal or could be something else wrong?</p>
<p>Thank you once again <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> <a class="mention" href="/u/tom">@tom</a></p>

username: tom
cooked: <p>It should be a lot closer if they were the same function. I think you would want to drop the softmax and use CrossEntropyLoss.</p>
<p>Best regards</p>
<p>Thomas</p>

username: peter_vala
cooked: <p>It worked! Everything gives the same values. Thank you very much.</p>