Transfering a model from keras to PyTorch

egor_nya: <p>I have a working model from this paper: <a href="https://arxiv.org/pdf/2006.08296.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2006.08296.pdf</a><br>
in keras that I’d like to re-write in with pytorch.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcb9479a5e60819c3cce62027a8375ad9a9981fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/dcb9479a5e60819c3cce62027a8375ad9a9981fc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dcb9479a5e60819c3cce62027a8375ad9a9981fc.png" alt="image" data-base62-sha1="vuC0d9WHIg2Xk9Hkrw3eKpOuw3O" width="448" height="500" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">725×808 17.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">class CaptchaSolver(nn.Module):
    def __init__(self):
        super(CaptchaSolver, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=1,
                               kernel_size=(5, 5), padding=[2, 2])
        self.dense1 = nn.Linear(67, 32)
        self.conv2 = nn.Conv2d(in_channels=1, out_channels=1,
                               kernel_size=(5, 5), padding=[2, 2])
        self.dense2 = nn.Linear(33*12, 48)
        
        self.conv3 = nn.Conv2d(in_channels=1, out_channels=1,
                               kernel_size=(5, 5), padding=[2, 2])
        self.dense3 = nn.Linear(16*8, 64)
        
        self.dense4 = nn.Linear(8*4, 512)
        self.drop = nn.Dropout2d(0.3)
        
        self.out_dense1 = nn.Linear(12288, 31)
        self.out_dense2 = nn.Linear(12288, 31)
        self.out_dense3 = nn.Linear(12288, 31)
        self.out_dense4 = nn.Linear(12288, 31)
        self.out_dense5 = nn.Linear(12288, 31)
        self.out_dense6 = nn.Linear(12288, 31)
        
        self.maxp = nn.MaxPool2d((2, 2))
        self.softm = nn.Softmax(dim=1)
        self.relu = nn.ReLU()
    def forward(self, x):
        x = self.conv1(x)
        x = self.dense1(x)
        x = self.relu(x)
        x = self.maxp(x)
        x = self.conv2(x)
        x = self.dense2(x)
        x = self.relu(x)
        x = self.maxp(x)
        x = self.conv3(x)
        x = self.dense3(x)
        x = self.relu(x)
        x = self.maxp(x)
        x = self.dense4(x)
        x = self.drop(x)
        x = torch.flatten(x)
        x1 = self.softm(self.out_dense1(x))
        x2 = self.softm(self.out_dense2(x))
        x3 = self.softm(self.out_dense3(x))
        x4 = self.softm(self.out_dense4(x))
        x5 = self.softm(self.out_dense5(x))
        x6 = self.softm(self.out_dense6(x))
        out = torch.cat([x1, x2, x3, x4, x5, x6])
        return out 
</code></pre>
<p>If I pass the last dimension of my tensor as <strong>in_features</strong> to the linear layer it changes it to <strong>out_features</strong> of the layer, and if I pass  67 * 25 * 1, which is the width * height * channels of generated captchas, it stops with a RuntimeError: mat1 dim 1 must match mat2 dim 0 error. I would like for it to work as close to my keras model as possible. What should I change?</p>

marco0410: <p>First, you have to change the ’<code>out_channels</code> of your conv layer in order to stay as close as possible to your keras model. E.g. <code>conv1</code> needs to have <code>out_channels=2</code>.</p>
<p>As a next step, your dense layers need to be implemented in another way. The first dense layer e.g. has 96 parameters which means</p>
<pre><code class="lang-auto">        self.dense1 = nn.Linear(2, 32)
</code></pre>
<p>The dense layer operates in the last dimension as far as I know, so in my opinion you would need something like</p>
<pre><code class="lang-auto"> x = x.view(1, -1, 2)
 x = self.dense1(x)
 x = x.view(1, 32, 25, 67)
</code></pre>
<p>with <code>batch_size = 1</code>. Or transposing would also be an option. But to be honest, I dont really know if this is the best solution or implementation, I just wanted to raise awareness for the differences between the given model and yours.</p>
<p>Best regards</p>

egor_nya: <p>I did something similar to what you described and it seems like it’s working, thanks!</p>