Differences of LSTM in Pytorch and Keras

username: juanko
cooked: <p>Hi,</p>
<p>I have been transforming a Keras-written code into a PyTorch code and encountered a problem that seems a bit tricky for me(Keras model <a href="https://github.com/zcakhaa/DeepLOB-Deep-Convolutional-Neural-Networks-for-Limit-Order-Books/blob/master/jupyter/run_train_tensorflow-version2.ipynb" rel="noopener nofollow ugc">here</a>). I’ve checked every single layer for Keras&amp;Pytorch for the model and before the LSTM layer, all of the outputs are same when the weights initialization are same for both of the framework(manually set it).</p>
<p>However, the LSTM layer acts differently even though the same weight is given for both of the models. Below is a code for Keras</p>
<pre><code class="lang-auto">conv_lstm = LSTM(64)(conv_reshape)
</code></pre>
<p>and below is the code for Pytorch</p>
<pre><code class="lang-auto">def __init__(...):
    ...
    self.LSTM = nn.LSTM(192, 64, 1, batch_first = True)
    ....

def forward(self, ...):
    ....
    lstm, _ = self.LSTM(output)
    lstm = lstm[:,-1,:]
    ....
</code></pre>
<p>If anyone has any idea about this, please help me out! Thanks in advance.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="juanko" data-post="1" data-topic="125616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanko/48/39916_2.png" class="avatar"> juanko:</div>
<blockquote>
<p>I’ve checked every single layer for Keras&amp;Pytorch for the model and before the LSTM layer, all of the outputs are same when the weights initialization are same for both of the framework(manually set it).</p>
</blockquote>
</aside>
<p>That sounds like a good approach!<br>
Could you share the code showing how you set the <code>nn.LSTM</code> parameters and compare the outputs?<br>
I guess that the parameter initialization might not have worked properly.</p>

username: juanko
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="125616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>parameters</p>
</blockquote>
</aside>
<p>Thanks for the reply ptrblck!</p>
<p>So, the code I wrote for matching the weights are as below,</p>
<pre><code class="lang-auto">model.LSTM.weight_ih_l0.data = torch.nn.Parameter(torch.FloatTensor(np.transpose(weights[28])).contiguous().to(device))
model.LSTM.weight_hh_l0.data = torch.nn.Parameter(torch.FloatTensor(np.transpose(weights[29])).contiguous().to(device))

model.LSTM.bias_ih_l0.data = torch.from_numpy(weights[30]).to(device)
model.LSTM.bias_hh_l0.data = torch.from_numpy(weights[30]).to(device)
</code></pre>
<p>Basically, the weights from Keras LSTM are in the list ‘weights’, and as Keras has only one bias(the same shape with both of the biases in the Pytorch LSTM), the same weights are given for both of the biases.</p>
<p>After running the code above, when I checked the weights for the model by running ‘model.LSTM.weight~ or model.LSTM.bias~’, all the weights seem to be properly set. If you have any idea where this difference of outputs coming from, please let me know!</p>
<p>Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="juanko" data-post="3" data-topic="125616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanko/48/39916_2.png" class="avatar"> juanko:</div>
<blockquote>
<p>and as Keras has only one bias(the same shape with both of the biases in the Pytorch LSTM), the same weights are given for both of the biases.</p>
</blockquote>
</aside>
<p>That doesn’t sound right. Could you check the shape of the <code>bias</code> in the Keras model and compare it to both <code>bias</code> parameters in the PyTorch implementation?</p>

username: Bam17
cooked: <p>This is true keras LSTM layer has only one bias while LSTM in torch has 2 biases. I faced such issue and thought to share it here to help people facing such issue. The way you are setting weights are correct but you need to set bias_hh_l0 as zero vector. To convert pretrained LSTM layer from keras to torch you need to do set up the weights in torch like such:</p>
<pre><code class="lang-auto">model.LSTM.weight_ih_l0.data = torch.nn.Parameter(torch.FloatTensor(np.transpose(weights[28])).contiguous().to(device))
model.LSTM.weight_hh_l0.data = torch.nn.Parameter(torch.FloatTensor(np.transpose(weights[29])).contiguous().to(device))

model.LSTM.bias_ih_l0.data = torch.from_numpy(weights[30]).to(device)
model.LSTM.bias_hh_l0.data = torch.from_numpy(np.zeros(weights[30].shape)).to(device)
</code></pre>