Convert LSTM from Keras to PyTorch

steven47: <p>Hi,</p>
<p>I’m trying to convert LSTM model code from Keras to Pytorch.</p>
<pre><code class="lang-auto"># input size
X_train.shape = (10000, 48)
# output size
y_train.shape = (10000, 16)
</code></pre>
<p>Here is the original keras model:</p>
<pre><code class="lang-auto">model = Sequential()
model.add(Embedding(16, 10, input_length=48))
model.add(CuDNNLSTM(50))
model.add(Dropout(0.1))
model.add(Dense(16, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
</code></pre>
<p>And this is my Pytorch model:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):

    def __init__(self, embedding_size = 10, hidden_size = 50, vocab_size = 16, tagset_size = 16, dropout_rate = 0.1):
        super().__init__()
        self.embedding_size = embedding_size # 10
        self.hidden_size = hidden_size # 50
        self.vocab_size = vocab_size # 16
        self.tagset_size = tagset_size # 16
        self.dropout_rate = dropout_rate # 0.1

        self.embedding = nn.Embedding(vocab_size, embedding_size) # (16, 10)
        self.lstm = nn.LSTM(embedding_size, hidden_size) # (10, 50)
        self.dropout = nn.Dropout(dropout_rate) # 0.1
        self.hidden2tag = nn.Linear(hidden_size, tagset_size) # (50, 16)

    def forward(self, x):
        embed = self.embedding(x)
        lstm_out, lstm_hidden = self.lstm(embed, None)
        lstm_out = lstm_out[:,-1,:]
        drop_out = self.dropout(lstm_out)
        output = self.hidden2tag(drop_out)
        return output

model = LSTMTagger(10, 50, 16, 16, 0.1)
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters())
</code></pre>
<p>I’m not sure whether I’m doing right or not. Any suggestion would be helpful.</p>

tom: <p>There are a few things where Keras has defaults that are different to PyTorch’s that you would want to aware of. Using the forum search function with <em>Keras LSTM</em>, you will find a number of threads on the subject, including the one below that might be a good starting point.</p>
<p>Best regards</p>
<p>Thomas</p>
<aside class="quote quote-modified" data-post="1" data-topic="13395">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/5f9b8f/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/im-stuck-in-converting-keras-lstm-to-pytorch-especially-in-data-type/13395">I'm stuck in converting Keras LSTM to Pytorch. Especially in Data Type</a> 
  </div>
  <blockquote>
    I’m trying to model converting from Keras to Pytorch. 
I think the problem is to train model part code. 
So, can anyone help this part for me. 
I’m newb at Pytorch… 
This is a Keras model configuration. 
The objective of this model is many to one model for prediction. 
The data ( [ a1 a2 a3 a4 a5] ) is fed into this model. And model predict a6. 
    model = Sequential()
    model.add(LSTM(10, return_sequences=True, input_shape=(windowsize,1)))
    model.add(LSTM(10))
    model.add(Dense(outputsi…
  </blockquote>
</aside>


steven47: <p>Thank you for your reply. I’m not sure whether my input size in each layer is correct or not. Suppose the size of the input data is (256,48). After passing through an embedding layer with 10 units, it will become (256,48,10). Now followed by an LSTM layer with 50 units, the size will be (256,48,50). However, the expected output shape should be (256,50), so I use <code>lstm_out = lstm_out[:,-1,:]</code> to adjust the size. Is this a reasonable adjustment? Thank you.</p>

tom: <p>So one thing you need to do to get it to work is to pass <code>batch_first</code> to the LSTM instantiation if that is what you want.</p>
<p>While taking the last timestep (as you do with <code>lstm_out[:, -1, :]</code>) is certainly a common way to set up sequence-to-one problems (assuming your inputs are of the same length), I would not call it a “size adjustment”. It says that the LSTM should “memorize” the relevant information. Other ways are possible, in fast.ai’s (Howard and Ruder) ULMFiT, they recommend to concatenate the last timestep with the average and max over time (but then the linear would have 3 * hidden_size inputs, of course).</p>
<p>Best regards</p>
<p>Thomas</p>