Convert keras model architecture to Pytorch

username: Ive_Xu
cooked: <p>I am trying to convert a keras model to pytorch for human activity recognition. The keras model could achieve up to 98% accuracy, while Pytorch model could achieve only ~60% accuracy. I couldn’t figure out the problem, first I was thinking of the padding=‘same’ of keras, but I have already adjusted the padding of pytorch already. Can you check what is wrong?</p>
<p>The keras code is as above</p>
<pre><code class="lang-auto">model = keras.Sequential()
model.add(layers.Input(shape=[100,12]))
model.add(layers.Conv1D(filters=32, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.Conv1D(filters=64, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.MaxPool1D(2))
model.add(layers.LSTM(64))
model.add(layers.Dense(units=128, activation='relu'))
model.add(layers.Dense(13, activation='softmax'))
model.summary()
</code></pre>
<p>and my pytorch model code is as below</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        seq_len = 100
        # output: [, 32, 100]
        self.conv1 = nn.Conv1d(seq_len, 32, kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm1d(32)
        # output: [, 64, 100]
        self.conv2 = nn.Conv1d(32, 64, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm1d(64)
        # output: [, 64, 50]
        self.mp = nn.MaxPool1d(kernel_size=2, stride=2)
        # output: [, 64]
        self.lstm = nn.LSTM(6, 64, 1)
        # output: [, 128]
        self.fc1 = nn.Linear(64, 128)
        # output: [, 13]
        self.fc2 = nn.Linear(128, 13)
        
    def forward(self, x):
        x = self.conv1(x) 
        x = self.bn1(x)
        x = F.relu(x)

        x = self.conv2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.mp(x)
        
        out, _ = self.lstm(x)
        x = out[:, -1, :]
        
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        
        return x
</code></pre>

username: Ive_Xu
cooked: <p>And this is the model summary of the Keras model, the output size of each layers do match.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/706687c27ca840d150e764f478ae442c69f77faf.png" data-download-href="https://discuss.pytorch.org/uploads/default/706687c27ca840d150e764f478ae442c69f77faf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/706687c27ca840d150e764f478ae442c69f77faf_2_648x500.png" alt="image" data-base62-sha1="g2l96z4y6J7hiYK5nYUrHvyiIFx" width="648" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/706687c27ca840d150e764f478ae442c69f77faf_2_648x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/706687c27ca840d150e764f478ae442c69f77faf_2_972x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/0/706687c27ca840d150e764f478ae442c69f77faf.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1182×912 52.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I guess the issue could be created by the <code>nn.LSTM</code> layer in the default setup, which would expect an input in the shape <code>[seq_len, batch_size, nb_features]</code> and could thus process the inputs in a wrong way.<br>
If you want to pass inputs with the batch dimension in <code>dim0</code>, you could create the modules with <code>batch_first=True</code>. The <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM">docs</a> explain the input shapes a bit more.</p>