Converting a Keras LSTM model to Pytorch

username: sushmit_roy
cooked: <p>I am trying to convert an LSTM &amp; Embedding model from Keras to Pytorch. The Keras model summary looks like this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93ad8e93ce7fe0976762f45779cc178b9ac8fcb8.png" data-download-href="https://discuss.pytorch.org/uploads/default/93ad8e93ce7fe0976762f45779cc178b9ac8fcb8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93ad8e93ce7fe0976762f45779cc178b9ac8fcb8_2_690x409.png" alt="image" data-base62-sha1="l4q1ZK37rCBps5fq5gzEtcqjbjG" width="690" height="409" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93ad8e93ce7fe0976762f45779cc178b9ac8fcb8_2_690x409.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93ad8e93ce7fe0976762f45779cc178b9ac8fcb8_2_1035x613.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/93ad8e93ce7fe0976762f45779cc178b9ac8fcb8_2_1380x818.png 2x" data-dominant-color="1C1C1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1838×1092 211 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.</p>
<p>The converted Pytorch model looks like this</p>
<blockquote>
<p>class test_model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(test_model,self).<strong>init</strong>()<br>
self.embed0 = nn.Embedding(12,emb_size(12))<br>
self.embed1 = nn.Embedding(53,emb_size(53))<br>
self.embed2 = nn.Embedding(12,emb_size(12))<br>
self.LSTM = nn.LSTM(16, 64,batch_first = True)<br>
self.linear = nn.Linear(64, 186)</p>
<pre><code>def forward(self,X):
    embed_0 = self.embed0(X[:,:,0])
    embed_1 = self.embed1(X[:,:,1])
    embed_2 = self.embed2(X[:,:,2])
    X = torch.cat((embed_0,embed_1,embed_2),dim=-1)
    output,hidden = LSTM(X) 
    return self.linear(output)
</code></pre>
</blockquote>
<p>However, I am getting a different loss. The dataset/optimizer and loss functions are same across Keras and Pytorch. Any idea ?</p>

username: ptrblck
cooked: <p>I’m not deeply familiar with reading the Keras output but are you using two <code>Dense</code> layers at the end?<br>
The <code>softmax (Dense)</code> output shape is given as <code>(None, 186, 12)</code> which doesn’t fit your <code>nn.Linear</code> output. If that’s the case, I’m unsure why a shape mismatch error isn’t raised when e.g. calculating the loss.</p>

username: HashRocketSyntax
cooked: <p>Do you need <code>nn.Softmax</code> at the end of your torch model?</p>
<pre><code class="lang-auto">        nn.Linear(..., num_classes),
        nn.Softmax(dim=1)
</code></pre>
<p>Torch and tf handle softmax differently. Take a look at these examples:</p>
<ul>
<li><a href="https://aiqc.readthedocs.io/en/latest/notebooks/keras_multi-label_classification.html" class="inline-onebox" rel="noopener nofollow ugc">Keras: Tabular Classify Multi-Label — AIQC</a></li>
<li><a href="https://aiqc.readthedocs.io/en/latest/notebooks/pytorch_multi-label_classification.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch: Tabular Classify Multi-Label — AIQC</a></li>
</ul>

username: sushmit_roy
cooked: <p>Thanks for the reply. I changed my module to look like this. The reshape at the end is to make sure it predicts 12 classes. I think the issue might be in training. Any idea if I am using the <code>cat</code> layer correctly and the reshape for crossentropy?</p>
<pre><code class="lang-auto">class test_model(nn.Module):
    def __init__(self):
        super(test_model,self).__init__()
        self.embed0 = nn.Embedding(12,emb_size(12))
        self.embed1 = nn.Embedding(53,emb_size(53))
        self.embed2 = nn.Embedding(12,emb_size(12))
        self.LSTM = nn.LSTM(16, 64,batch_first = True)
        self.linear1 = nn.Linear(64, 64)
        self.linear2 = nn.Linear(64, 12)

        
    def forward(self,X):
        
        embed_0 = self.embed0(X[:,:,0])
        embed_1 = self.embed1(X[:,:,1])
        embed_2 = self.embed2(X[:,:,2])
        X = torch.cat((embed_0,embed_1,embed_2),dim=-1)
        output,hidden = LSTM(X) 
        X = self.linear1(output)
        X = self.linear2(X)
        return X.reshape(-1,12,186)
</code></pre>
<p>also do you happen to know if keras equivalent for<br>
<code>LSTM(memory_units, return_sequences=True, stateful=False, name='lstm')</code><br>
would be<br>
<code>nn.LSTM(16, 64,batch_first = True)</code><br>
and the use the <code>output</code></p>

username: sushmit_roy
cooked: <p>You dont need to use softmax if we are using CrossEntropy loss. <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.10 documentation</a></p>

username: ptrblck
cooked: <p>I would recommend to check the shape of <code>X</code> before the <code>reshape</code> operation, since flattening the data into the batch dimension is usually wrong as done via <code>X.reshape(-1,12,186)</code>.<br>
If you are not careful, you could change the batch size of <code>X</code> which would then create shape mismatches e.g when trying to calculate the loss.</p>

username: sushmit_roy
cooked: <p>Thanks without the reshape my size is <code>torch.Size([32, 186, 12])</code> if I pass this to loss (Crossentropy) it gives me an error. Basically, the # of classes is 12 ? if I follow the Crossentropy doc I need to change it <code>(N,C,d1)</code></p>

username: ptrblck
cooked: <p>Yes, your interpretation of the expected input shape is correct.<br>
However, the <code>reshape</code> operation is wrong as it would interleave the values.<br>
Since you want to swap some dimensions, use <code>X = X.permute(0, 2, 1).contiguous()</code> instead.</p>