Rebuild network class from tensorflow

makkkiiii: <p>Hi guys</p>
<p>I am currently trying to rebuild a network from tensoflow in pytorch. The library onnx2pytorch worked for me, but I would like to build a replica class from the same network in pytroch. The idea is to use this class in order to test inference with different number systems like bfloat or posit.</p>
<p>What I did so far:</p>
<ul>
<li>Converting tf model to pytroch</li>
</ul>
<pre><code class="lang-auto">from onnx2pytorch import ConvertModel

os.environ['TF_KERAS'] = '1'
pretrained_model = tf_load_model(config['model_list']['dna821_fp32'])
onnx_model = onnxmltools.convert_keras(pretrained_model)
pretrained_model = ConvertModel(onnx_model, experimental=True)

    
</code></pre>
<p>This leads to the following model which has the same accuracy and size as in tensorflow:</p>
<pre><code class="lang-auto">ConvertModel(
  (Transpose_model/conv2d/BiasAdd__29:0): Transpose()
  (Conv_model/conv2d/BiasAdd:0): Conv2d(3, 8, kernel_size=(7, 7), stride=(1, 1))
  (Relu_model/conv2d/Relu:0): ReLU(inplace=True)
  (MaxPool_model/max_pooling2d/MaxPool:0): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
  (Conv_model/conv2d_1/BiasAdd:0): Conv2d(8, 150, kernel_size=(4, 4), stride=(1, 1))
  (Relu_model/conv2d_1/Relu:0): ReLU(inplace=True)
  (MaxPool_model/max_pooling2d_1/MaxPool:0): MaxPool2d(kernel_size=(6, 6), stride=(6, 6), padding=0, dilation=1, ceil_mode=False)
  (Transpose_model/max_pooling2d_1/MaxPool__43:0): Transpose()
  (Reshape_model/flatten/Reshape:0): Reshape(shape=[  -1 1350])
  (MatMul_model/dense/BiasAdd:0): Linear(in_features=1350, out_features=340, bias=True)
  (Relu_model/dense/Relu:0): ReLU(inplace=True)
  (MatMul_model/dense_1/BiasAdd:0): Linear(in_features=340, out_features=490, bias=True)
  (Relu_model/dense_1/Relu:0): ReLU(inplace=True)
  (MatMul_model/dense_2/BiasAdd:0): Linear(in_features=490, out_features=43, bias=True)
  (Softmax_dense_2): Softmax(dim=-1)
)
</code></pre>
<ul>
<li>I build with this information my model class in pytroch</li>
</ul>
<pre><code class="lang-auto">import torch.nn as nn


class GTRSB(nn.Module):
    def __init__(self):
        super().__init__()
        self.seq = nn.Sequential(

            nn.Conv2d(in_channels=3, out_channels=8, kernel_size=(7, 7), stride=(1, 1)),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False),

            nn.Conv2d(in_channels=8, out_channels=150, kernel_size=(4, 4), stride=(1, 1)),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(6, 6), stride=(6, 6), padding=0, dilation=1, ceil_mode=False),

            nn.Flatten(),
            nn.Linear(in_features=1350, out_features=340, bias=True),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=340, out_features=490, bias=True),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=490, out_features=43, bias=True),
            nn.Softmax(dim=-1)

        )

    def forward(self, x):
        return self.seq(x)
</code></pre>
<ul>
<li>I initialized the GTRSB  model and copied the parameters from the converted model to my model which works without errors:</li>
</ul>
<pre><code class="lang-auto">pretrained_layer_name = [name for name, param in pretrained_model.named_parameters()]
pretrained_state = pretrained_model.state_dict()

model = GTRSB()
model_state_dict = model.state_dict()
model_layer_name = [name for name, param in model.named_parameters()]
for i in range(len(model_state_dict)):
	model_state_dict[model_layer_name[i]] = pretrained_state[pretrained_layer_name[i]]
model.load_state_dict(model_state_dict)
</code></pre>
<p>leads to:</p>
<pre><code class="lang-auto">GTRSB(
  (seq): Sequential(
    (0): Conv2d(3, 8, kernel_size=(7, 7), stride=(1, 1))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(8, 150, kernel_size=(4, 4), stride=(1, 1))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=(6, 6), stride=(6, 6), padding=0, dilation=1, ceil_mode=False)
    (6): Flatten(start_dim=1, end_dim=-1)
    (7): Linear(in_features=1350, out_features=340, bias=True)
    (8): ReLU(inplace=True)
    (9): Linear(in_features=340, out_features=490, bias=True)
    (10): ReLU(inplace=True)
    (11): Linear(in_features=490, out_features=43, bias=True)
    (12): Softmax(dim=-1)
  )
)

</code></pre>
<p><strong>Problems</strong><br>
My network (model) somehow could not take the input shape as the converted one (pretrained_model). I have to change my matrix shape from (12630, 48, 48, 3) to (12630, 3, 48, 48) in order to do the inference. Furthermore, the performance is not the same. My network has a much lower accuracy then the converted one.</p>
<p>I assume that I have some errors in my GTRSB class.<br>
The converted network has a few layer more in the descrption then my network. As an example there is the</p>
<pre><code class="lang-auto">(Transpose_model/conv2d/BiasAdd__29:0): Transpose()
</code></pre>
<p>layer (or I think at least it is one) in the begin of the summary. The same layer shows up in the middle of the summary again. I think this could lead to the behavior. Does someone of you have an idea what the error could be or how to fix my model class?</p>
<p>Thanks a lot</p>

J_Johnson: <p>Pytorch conv layers take [batch_size, classes, width, height]. If your images are [batch_size, width, height, classes], you will need to permute.</p>
<p>Also, you need the images to be in the range of 0.0, 1.0. A common method to convert from PIL or NumPy to tensor is using the <code>transforms.ToTensor()</code> class found here:</p>
<p><a href="https://pytorch.org/vision/stable/transforms.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/transforms.html</a></p>
<p>That will both permute and scale the image and convert to tensor.</p>

makkkiiii: <p>Hi Johnson</p>
<p>Thank you for the comment. I haven’t used permute, but torch.view. I changed this part to permute (You are right it is better to use permute). I also tried to transform and convert the images with the <code>transforms.ToTensor()</code> class. But still no changes in the accuracy after all.</p>
<p>I think it is still something wrong with the build class.</p>

J_Johnson: <p>If you use <code>transforms.ToTensor()</code> there is no need to permute.</p>
<p>Can you share your training script?</p>

makkkiiii: <p>I found the bug. It was in the forward function. I am not sure why but converting models from tensorflow to pycharm adds some transpose between the hidden layers. In my case before the first conv and the last maxpool. (before flatten)</p>
<p>The correct class with the fix in the forward function is:</p>
<pre><code class="lang-auto">class GTRSB(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv_0 = nn.Sequential(

            nn.Conv2d(in_channels=3, out_channels=8, kernel_size=(7, 7), stride=(1, 1)),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False),
        )

        self.conv_1 = nn.Sequential(

            nn.Conv2d(in_channels=8, out_channels=150, kernel_size=(4, 4), stride=(1, 1)),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(6, 6), stride=(6, 6), padding=0, dilation=1, ceil_mode=False),

        )

        self.dense_0 = nn.Sequential(

            nn.Flatten(),
            nn.Linear(in_features=1350, out_features=340, bias=True),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=340, out_features=490, bias=True),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=490, out_features=43, bias=True),
            nn.Softmax(dim=-1)

        )

    def forward(self, x):
        x = torch.transpose(x, 1, 2)
        x = torch.transpose(x, 1, 3)
        x = self.conv_0(x)
        x = self.conv_1(x)
        x = torch.transpose(x, 1, 2)
        x = torch.transpose(x, 2, 3)
        x = self.dense_0(x)
        return x

</code></pre>