Conv2D Keras to PyTorch

RobinFrcd: <p>Hi,<br>
I’m trying to convert a custom UNET implementation from Tensorflow to PyTorch. I’ve encountered some problems with the Conv2D layers.</p>
<p>I know there could be some trouble with padding, it tried <a href="https://stackoverflow.com/a/68171022/5692012" rel="noopener nofollow ugc">this</a> and <a href="https://stackoverflow.com/questions/66293554/translate-conv2d-from-pytorch-code-to-tensorflow" rel="noopener nofollow ugc">this</a> but it didn’t help.</p>
<p>My conversion code looks like this:</p>
<pre><code class="lang-python">from keras.layers import Conv2D
from torch import nn

import torch
import pandas as pd
import numpy as np

img = np.random.rand(1, 256, 256, 1)

## TF Init
conv_tf = Conv2D(
    64, 3, activation="relu", padding="same", kernel_initializer="he_normal", 
)

conv_tf(img)
conv_tf.bias = np.random.rand(64)

## PT Init + copy weights
conv_torch = nn.Conv2d(
    in_channels=1, out_channels=64, kernel_size=3, padding="same", bias=True
)
conv_torch.weight = nn.parameter.Parameter(
    torch.Tensor(conv_tf.weights[0].numpy().transpose(3, 2, 0, 1))
)
conv_torch.bias = nn.parameter.Parameter(torch.Tensor(conv_tf.bias))

conv_torch = nn.Sequential(
    conv_torch,
    nn.ReLU()
)
</code></pre>
<p>If I try to run a tensor through the models, the result is close but not the same (with huge differences between a few points)</p>
<pre><code class="lang-python">pred_tf = conv_tf(img).numpy()
pred_pt = conv_torch(torch.Tensor(img).reshape(1, 1, 256, 256)).detach().numpy().reshape(pred_tf.shape)

pred_tf.mean()
#0.7202551

pred_pt.mean()
#0.7202549
</code></pre>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:left"></th>
<th style="text-align:right">TF - PT</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">count</td>
<td style="text-align:right">4.1943e+06</td>
</tr>
<tr>
<td style="text-align:left">mean</td>
<td style="text-align:right">-2.2992e-09</td>
</tr>
<tr>
<td style="text-align:left">std</td>
<td style="text-align:right">0.969716</td>
</tr>
<tr>
<td style="text-align:left">min</td>
<td style="text-align:right">-3.85477</td>
</tr>
<tr>
<td style="text-align:left">25%</td>
<td style="text-align:right">-0.641259</td>
</tr>
<tr>
<td style="text-align:left">50%</td>
<td style="text-align:right">0</td>
</tr>
<tr>
<td style="text-align:left">75%</td>
<td style="text-align:right">0.641266</td>
</tr>
<tr>
<td style="text-align:left">max</td>
<td style="text-align:right">3.8742</td>
</tr>
</tbody>
</table>
</div><p>Thanks</p>

KFrank: <p>Hi Robin!</p>
<aside class="quote no-group quote-modified" data-username="RobinFrcd" data-post="1" data-topic="157704" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/robinfrcd/48/51851_2.png" class="avatar"> RobinFrcd:</div>
<blockquote>
<p>If I try to run a tensor through the models, the result is close but not the same (with huge differences between a few points)</p>
<pre data-code-wrap="python"><code class="lang-python">pred_pt = conv_torch(torch.Tensor(img).reshape(1, 1, 256, 256)).detach().numpy().reshape(pred_tf.shape)
</code></pre>
</blockquote>
</aside>
<p>The problem is that <code>reshape()</code> doesn’t really do what you think it is doing<br>
and is messing up the two-dimensional structure of the image.</p>
<p>Use <code>permute()</code> on the pytorch side and <code>transpose()</code> on the numpy side.</p>
<p>Here is a slightly tweaked version of your code with the fix for <code>reshape()</code><br>
at the end:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import keras
2022-07-28 23:46:08.455782: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2022-07-28 23:46:08.455958: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
&gt;&gt;&gt; keras.__version__
'2.9.0'
&gt;&gt;&gt;
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0.dev20220727'
&gt;&gt;&gt;
&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; np.__version__
'1.23.1'
&gt;&gt;&gt;
&gt;&gt;&gt; from keras.layers import Conv2D
&gt;&gt;&gt; from torch import nn
&gt;&gt;&gt;
&gt;&gt;&gt; keras.utils.set_random_seed (2022)
&gt;&gt;&gt; np.random.seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; img = np.random.rand(1, 256, 256, 1)
&gt;&gt;&gt;
&gt;&gt;&gt; ## TF Init
&gt;&gt;&gt; conv_tf = Conv2D(
...     64, 3, activation="relu", padding="same", kernel_initializer="he_normal",
... )
&gt;&gt;&gt;
&gt;&gt;&gt; _ = conv_tf(img)
2022-07-28 23:46:17.830862: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2022-07-28 23:46:17.831780: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2022-07-28 23:46:17.832975: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2022-07-28 23:46:18.957824: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusolver64_11.dll'; dlerror: cusolver64_11.dll not found
2022-07-28 23:46:18.958813: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2022-07-28 23:46:18.959940: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2022-07-28 23:46:18.960028: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-07-28 23:46:18.960743: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
&gt;&gt;&gt; conv_tf.bias = np.random.rand(64)
&gt;&gt;&gt;
&gt;&gt;&gt; ## PT Init + copy weights
&gt;&gt;&gt; conv_torch = nn.Conv2d(
...     in_channels=1, out_channels=64, kernel_size=3, padding="same", bias=True
... )
&gt;&gt;&gt; conv_torch.weight = nn.parameter.Parameter(
...     torch.Tensor(conv_tf.weights[0].numpy().transpose(3, 2, 0, 1))
... )
&gt;&gt;&gt; conv_torch.bias = nn.parameter.Parameter(torch.Tensor(conv_tf.bias))
&gt;&gt;&gt;
&gt;&gt;&gt; conv_torch = nn.Sequential(
...     conv_torch,
...     nn.ReLU()
... )
&gt;&gt;&gt;
&gt;&gt;&gt; pred_tf = conv_tf(img).numpy()
&gt;&gt;&gt; pred_pt = conv_torch(torch.Tensor(img).reshape(1, 1, 256, 256)).detach().numpy().reshape(pred_tf.shape)
&gt;&gt;&gt; pred_pt_fix = conv_torch(torch.Tensor(img).permute (0, 3, 1, 2)).detach().numpy().transpose (0, 2, 3, 1)
&gt;&gt;&gt;
&gt;&gt;&gt; np.abs (pred_tf - pred_pt).max()       # reshape ruins the two-dimensional structure of the image
4.251324
&gt;&gt;&gt; np.abs (pred_tf - pred_pt_fix).max()   # using permute / transpose fixes the problem
9.536743e-07
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

RobinFrcd: <p>Now I’ll know that <code>.reshape</code> is not magic and should be used with care ! Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>