CNN-LSTM problem

Nassim_Mokhtari: <p>Hi,<br>
I have implemented a hybdrid model with CNN &amp; LSTM in both Keras and PyTorch, the network is composed by 4 layers of convolution with an output size of 64 and a kernel size of 5, followed by 2 LSTM layer with 128 hidden states, and then a Dense layer of 6 outputs for the classification.<br>
In fact, i have juste implemented the DeepConvLSTM proposed here <a href="https://www.researchgate.net/publication/291172413_Deep_Convolutional_and_LSTM_Recurrent_Neural_Networks_for_Multimodal_Wearable_Activity_Recognition" rel="nofollow noopener">https://www.researchgate.net/publication/291172413_Deep_Convolutional_and_LSTM_Recurrent_Neural_Networks_for_Multimodal_Wearable_Activity_Recognition</a>.<br>
My problem is with PyTorch version, i’m getting around 18~19 % of accuracy, while Keras is giving 86~87%. I don’t understand why,  i’m using the same parameters for both networks and the same optimizer RMSROP.<br>
I also tried to use GRU instead of LSTM, but getting the same problem, seems like there is a probleme with the hybridation its selfs, but i can not figure it out.<br>
here is my scripts</p>
<p><strong>Keras version :</strong></p>
<pre><code class="lang-auto">type or paste cdef ConvLSTM_Keras(input_shape):
    from keras.models import Sequential
    from keras.layers import Dense,Conv1D,LSTM
    model = Sequential()
    model.add(Conv1D(64, 5,
                     activation='relu',
                     input_shape=input_shape))
    model.add(Conv1D(64, 5, activation='relu'))
    model.add(Conv1D(64, 5, activation='relu'))
    model.add(Conv1D(64, 5, activation='relu'))
    model.add(LSTM(128,return_sequences=True))
    model.add(LSTM(128,return_sequences=False))
    model.add(Dense(6, activation='softmax'))
    return model

model.compile(loss=keras.losses.categorical_crossentropy,
                  optimizer=keras.optimizers.rmsprop(
                      learning_rate=0.001
                  ),
                  metrics=['accuracy'])

model.fit(x_train, y_train,
              epochs=20,
              batch_size=100,
              verbose=1,
              validation_data=(x_val, y_val))
</code></pre>
<p><strong>PyTorch Version</strong></p>
<pre><code class="lang-auto"> def __init__(self):
        super(SimpleCNN, self).__init__()
        self.conv1 = torch.nn.Conv1d(in_channels=1, out_channels=64, kernel_size=5)
        self.conv2 = torch.nn.Conv1d(in_channels=64,out_channels=64, kernel_size=5)
        self.conv3 = torch.nn.Conv1d(in_channels=64, out_channels=64, kernel_size=5)
        self.conv4 = torch.nn.Conv1d(in_channels=64, out_channels=64, kernel_size=5)
        self.lstm1 = torch.nn.LSTM(
            input_size= 545,
            hidden_size=128,
            num_layers=2,
        )
        self.fc2 = torch.nn.Linear(128, 6)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
        x,_ = self.lstm1(x)
        x = x[:, -1, :]
        x = self.fc2(x)
        return (x)

import torch.optim as optim
def createLossAndOptimizer(net, learning_rate=0.001):
    # Loss function
    loss = torch.nn.CrossEntropyLoss()

    # Optimizer
    optimizer = optim.rmsprop(net.parameters(), lr=learning_rate)
    return (loss, optimizer)
</code></pre>
<p>Hope you can help me, thanks.<br>
Nassim</p>

ptrblck: <p>You could try to get matching results using a few layers (or blocks) and then scaling it up.<br>
E.g. I couldn’t find a lot about how the stacked <code>LSTM</code> layers work, i.e. how is their hidden and state tensor defined, so this would be my starting point. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Nassim_Mokhtari: <p>Thanks for your answer, i’ll try it.</p>

G.M: <p>R u trying to use the last hidden state of LSTM only?</p>

Nassim_Mokhtari: <p>Yes, i’m doing it at  x = x[:,-1,:]</p>

G.M: <p>Alright, I believe here’s the problem. In ur model, if I assume the input to CNN is of shape [B, 1, L], then the CNN outputs a tensor of shape [B, C, L] where B is the batch size, L is the sequence length, C is the channel. You then fed it into LSTM hoping it would learn the temporal information of dimension 2(L). But what’s actually happening is that, according to <a href="https://pytorch.org/docs/1.0.1/nn.html#recurrent-layers" rel="nofollow noopener">here</a>, the LSTM by default assumes the input is of shape [L, B, INPUT_SIZE], so in this case, the LSTM is trying to learn the temporal between dimension 0(B), which apparently are independent with each other. That’s why the network is acting as a random number generator.<br>
To solve this problem, u would have to transpose ur tensor accordingly to match the expected input shape of the LSTM and get the last hidden state according to the output shape of LSTM.</p>

Nassim_Mokhtari: <p>Thanks for your help, i’ll try it, thanks a lot</p>

Nassim_Mokhtari: <p>Sorry for taking a long time to feedback, i was busy with another task, i tried your proposition and now it works perfectly, i’m having matching results between Keras and PyTorch, thanks !<br>
I have another question if you can answer me, i noticed that the pytorch’s implementations of RNNs (LSTM and GRU) are faster then Keras, and acheive better results, do you know why ?</p>

G.M: <p>I’m not familiar with keras so that’s out of my  reach. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

Berna_Yilmaz: <aside class="quote no-group" data-username="Nassim_Mokhtari" data-post="1" data-topic="69344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nassim_mokhtari/48/16927_2.png" class="avatar"> Nassim_Mokhtari:</div>
<blockquote>
<pre><code class="lang-auto">x = x[:, -1, :]
        x = self.fc2(x)
</code></pre>
</blockquote>
</aside>
<p><a class="mention" href="/u/g.m">@G.M</a> Hi,<br>
Can you share right code in pytorch ?<br>
Thanks.</p>

DrDumbenstein: <p>Really helpful thank you.</p>

NancyShehata: <p>Hi Nassim,<br>
I am facing the same problem of transposing the tensor, can you please share your code after working and transposing the tensor.<br>
Thank you</p>