TF to PT weight tranfer doesnt get same eval results

peter_vala: <p>Hey there,<br>
I’m trying to transfer weights between pytorch and tensorflow and viceversa. The problem that I’m getting is that even having the same weights for all layers the results are different for evaluation of the model and i can’t understand why.</p>
<p>this is my tensorflow model:</p>
<p>num_classes = 10<br>
input_shape = (28, 28, 1)<br>
tf.random.set_seed(2)<br>
model = keras.Sequential(<br>
[<br>
keras.Input(shape=input_shape),<br>
layers.Conv2D(4, kernel_size=(3, 3), name = “conv2d”),<br>
layers.MaxPooling2D(pool_size=(2, 2)),<br>
layers.Flatten(name=“flatten”),<br>
layers.Dense(num_classes, activation=“softmax”,name=“dense”),<br>
]<br>
)</p>
<p>and this is my pytorch model</p>
<p>torch.manual_seed(2)<br>
class Network(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv2d = nn.Conv2d(1,4,kernel_size=(3,3))<br>
self.hidden = nn.MaxPool2d((2,2))<br>
self.dense = nn.Linear(676, 10)<br>
self.softmax = nn.Softmax(dim=1)</p>
<pre><code>def forward(self, x):
    in_size = x.size(0)
    x = self.hidden(self.conv2d(x))
    x = x.view(in_size, -1)
    x = self.dense(x)
    return self.softmax(x)
</code></pre>
<p>I also test it after training tensorflow model but even after the weight the tranfer the results are completly different:</p>
<p>Tensorflow acc:  Test accuracy: 0.9228000044822693<br>
Pytorch acc : Test set: Average loss: 0.0013, Accuracy: 972/10000 (10%)</p>
<p>These are the weights also after the training:<br>
Tensorflow :<br>
[array([[[[ 0.46365955, -0.12814473,  0.14183734, -0.5937301 ]],</p>
<pre><code>     [[ 0.35576758,  0.5139991 ,  0.6756656 , -0.65329105]],

     [[ 0.48599064,  0.57487786, -0.11121935, -0.6914515 ]]],


    [[[ 0.27835783,  0.43169427,  0.33743912, -0.7600264 ]],

     [[ 0.547942  ,  1.0308589 ,  0.489442  , -0.67891765]],

     [[ 0.5344402 ,  0.6332875 ,  0.6738143 , -0.43144146]]],


    [[[-0.05558264,  0.7325994 ,  0.28004876, -0.17341942]],

     [[-0.1670951 ,  0.7585311 ,  0.54995114, -0.36898494]],

     [[ 0.3589517 ,  0.14311437,  0.41515586, -0.5163795 ]]]],
   dtype=float32),
</code></pre>
<p>array([ 0.27913818, -0.39000487, -0.23639436, -0.2614709 ], dtype=float32)]</p>
<p>Pytorch :<br>
Parameter containing:<br>
tensor([[[[ 0.4637,  0.3558,  0.4860],<br>
[ 0.2784,  0.5479,  0.5344],<br>
[-0.0556, -0.1671,  0.3590]]],</p>
<pre><code>    [[[-0.1281,  0.5140,  0.5749],
      [ 0.4317,  1.0309,  0.6333],
      [ 0.7326,  0.7585,  0.1431]]],


    [[[ 0.1418,  0.6757, -0.1112],
      [ 0.3374,  0.4894,  0.6738],
      [ 0.2800,  0.5500,  0.4152]]],


    [[[-0.5937, -0.6533, -0.6915],
      [-0.7600, -0.6789, -0.4314],
      [-0.1734, -0.3690, -0.5164]]]], requires_grad=True)
</code></pre>
<p>Parameter containing:<br>
tensor([ 0.2791, -0.3900, -0.2364, -0.2615], requires_grad=True)</p>
<p>Can someone give me some light on what is creating this big difference?</p>

ptrblck: <p>I don’t know how you’ve loaded the parameters from TF to PyTorch, but would recommend to check the layers in isolation and make sure e.g. the memory layout, flipping etc. are right.<br>
In your PyTorch model you re using <code>nn.Softmax</code> as the last activation, which is usually wrong. Guessing you are working on a multi-class classification using <code>nn.CrossEntropyLoss</code>, remove the softmax as the criterion expect raw logits.</p>

peter_vala: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your answer. I’m actually new and i’m still learning this concepts.<br>
I did remove the softmax activation from the pytorch net and didn’t change anything.<br>
This is the code I’m using to transfer weights from keras to pytorch its the code that i found in the github:</p>
<p>def keras_to_pyt(km, pm):<br>
weight_dict = OrderedDict()<br>
for layer in km.layers:<br>
if type(layer) is keras.layers.Conv2D:<br>
weight_dict[layer.get_config()[‘name’] + ‘.weight’] = np.transpose(layer.get_weights()[0], (3, 2, 0, 1))<br>
print(layer.get_config()[‘name’])<br>
<span class="hashtag">#print</span>(weight_dict[layer.get_config()[‘name’] + ‘.weight’])<br>
weight_dict[layer.get_config()[‘name’] + ‘.bias’] = layer.get_weights()[1]<br>
elif type(layer) is keras.layers.Dense:<br>
weight_dict[layer.get_config()[‘name’] + ‘.weight’] = np.transpose(layer.get_weights()[0], (1, 0))<br>
weight_dict[layer.get_config()[‘name’] + ‘.bias’] = layer.get_weights()[1]<br>
pyt_state_dict = pm.state_dict()<br>
<span class="hashtag">#print</span>(pyt_state_dict)<br>
<span class="hashtag">#print</span>(weight_dict)<br>
print(pyt_state_dict.keys())<br>
for key in pyt_state_dict.keys():<br>
print(key)<br>
if “flatten” in key:<br>
print(“flatten layer no weights or biases”)<br>
continue</p>
<pre><code>    key1 = remove_prefix(key,"layers.")
    pyt_state_dict[key] = torch.from_numpy(weight_dict[key1])
    #print(pyt_state_dict[key])
pm.load_state_dict(pyt_state_dict)
return pm</code></pre>