Migrating from keras to pytorch

Mary_nj: <p>hi every body. please help me.<br>
in just want to transfer a code from keras to pytorch<br>
first q: is this two models are same? if not what is exact equivalence code in pytorch:)<br>
so:<br>
is this code…?</p>
<pre><code class="lang-auto">  question = layers.Input(shape=(None,), dtype='int32', name='question')
        answer = layers.Input(shape=(None,), dtype='int32', name='answer')

        embedding = layers.Embedding(
            mask_zero=True,
            input_dim=self.vocab_size,
            output_dim=self.embedding_size,
            weights=[self.w2v_weights],
            trainable=True
        )

        lstm_1 = layers.LSTM(units=80, return_sequences=True)
        lstm_2 = layers.LSTM(units=80, return_sequences=False)
        question_model = lstm_2(lstm_1(embedding(question)))
        answer_model = lstm_2(lstm_1(embedding(answer)))

        sim = layers.dot([question_model, answer_model], axes=1, normalize=True)
        #     sim = layers.Activation(activation='sigmoid')(sim)
        qa_similary_model = models.Model(
            inputs=[question, answer],
            outputs=[sim],
        )
        qa_similary_model.compile(
            loss='mean_squared_error', optimizer='nadam', metrics=['accuracy'])

        embedding_model = models.Model(
            inputs=[question],
            outputs=[question_model]
        )
        return qa_similary_model, embedding_model
</code></pre>
<p>is equivalent to this?</p>
<pre><code class="lang-auto">   class LSTM_Model(Model):
        """
        a class to define multiple models
        """

        def __init__(self, **kwargs):
            super().__init__(**kwargs)

        def forward(self, question, answer):
            question_embedding = self.embeddings(question)
            # print("question embedding shape:", question_embedding.shape)
            answer_embedding = self.embeddings(answer)
            # print("answer embedding shape:", answer_embedding.shape)
            q_output, (qhidden, qcell) = self.lstm(question_embedding)
            # print("q_output shape:", q_output.shape)
            # print("qhidden shape:", qhidden.shape)
            # print("qcell shape:", qcell.shape)
            a_output, (ahidden, acell) = self.lstm(answer_embedding)
            # print("a_output shape:", a_output.shape)
            # print("ahidden shape:", ahidden.shape)
            # print("acell shape:", acell.shape)
            # qa_similary = torch.mm(qhidden[-1], ahidden[-1])
            # qa_similary =torch.matmul((qhidden[-1]), torc.th(ahidden[-1]))
            qa_similary = torch.matmul((q_output[-1]), torch.t(a_output[-1]), )
            # print("qa_similary shape:", qa_similary.shape)
            return qa_similary, qhidden
</code></pre>
<p>i need help <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

ptrblck: <p>You didn’t define any modules in your PyTorch code, so it shouldn’t be executable (or you didn’t post this code snippet).<br>
Also, in Keras you are using two LSTM layers and pass the output of the first one to the second, while you are only using one in PyTorch, so you might want to add the second one.</p>

Mary_nj: <p>tnx for you answering,yes i forget to copy it here, all code’s model is here:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()

        self.embeddings = nn.Embedding(num_embeddings=kwargs["vocab_size"],
                                      embedding_dim=kwargs["embedding_dim"],
                                      padding_idx=kwargs["pad_idx"])
        self.embeddings.weight.requires_grad = True  # to not refine-tune

        if kwargs["model"] == "lstm":
            self.lstm = nn.LSTM(input_size=kwargs["embedding_dim"],  # input
                                hidden_size=kwargs["lstm_units"],  # output
                                num_layers=kwargs["lstm_layers"],
                                bidirectional=False,
                                batch_first=True)
        if kwargs["model"] == "BiLSTM":
            self.lstm = nn.LSTM(input_size=kwargs["embedding_dim"],  # input
                                hidden_size=kwargs["bilstm_units"],  # output
                                num_layers=kwargs["bilstm_layers"],
                                bidirectional=True,
                                batch_first=True)

        self.dropout = nn.Dropout(kwargs["dropout"])
        self.tanh = F.tanh
        self.dropout = nn.Dropout(kwargs["dropout"])

    def forward(self):
        pass


class LSTM_Model(Model):
    """
    a class to define multiple models
    """
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def forward(self, question, answer):
        question_embedding = self.embeddings(question)
        # print("question embedding shape:", question_embedding.shape)
        answer_embedding = self.embeddings(answer)
        # print("answer embedding shape:", answer_embedding.shape)
        q_output, (qhidden, qcell) = self.lstm(question_embedding)
        print("q_output shape:", q_output.shape)
        # print("qhidden shape:", qhidden.shape)
        # print("qcell shape:", qcell.shape)
        a_output, (ahidden, acell) = self.lstm(answer_embedding)
        print("a_output shape:", a_output.shape)
        # print("ahidden shape:", ahidden.shape)
        # print("acell shape:", acell.shape)
        # qa_similary = torch.mm(qhidden[-1], ahidden[-1])
        # qa_similary =torch.matmul((qhidden[-1]), torc.th(ahidden[-1]))
        q_output = q_output[-1]
        q_output = q_output.squeeze()
        a_output = a_output[-1]
        a_output = a_output.squeeze()
        mm = torch.mul((q_output), (a_output))
        mm -= mm.min(1, keepdim=True)[0]
        mm /= mm.max(1, keepdim=True)[0]
        qa_similary =torch.mean(mm, dim=1)
        # print("qa_similary shape:", qa_similary.shape)
        return qa_similary, qhidden

    print("**************************MODEL DEFINE &amp; CREATED!****************************")

</code></pre>
<p>would you please say it work exactly as it is in keras?</p>

ptrblck: <p>I don’t think it will work exactly as in Keras, as it seems you are either still missing the second <code>nn.LSTM</code>.<br>
I’m not deeply familiar with Keras, but I thought there is a <code>Bidirectional</code> layer, in case you want to use a bidirectional LSTM.<br>
However, in the current Keras snippet, the two LSTM modules are piped to each other.<br>
Could you explain a bit, how the Keras model works and how the lstms are interacting with each other?</p>

Mary_nj: <p>I did a question matching task in keras, it has two input for two similar question.<br>
Lets named them q,q’<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I have two lstm that q and q’ are  passing from both of them,in this way:<br>
q-----&gt;embedding-----&gt;lstm1-----&gt;lstm2—&gt;result:sum_a<br>
q’-----&gt;embedding-----&gt;lstm1-----&gt;lstm2—&gt;result:sum_b<br>
Ok?<br>
So then the models output should be two arg:</p>
<ol>
<li>the q’ result (sum_a)</li>
<li>Dot product of sum_a and sum_b</li>
</ol>
<p>So it is the explanation in LSTM term of keras…</p>
<p>And:<br>
In second BiLSTM model we have just one bilstm layer ofter embedding</p>
<p>q-----&gt;embedding-----&gt;bilstm—&gt;result:sum_a<br>
q’-----&gt;embedding-----&gt;bilstm1—&gt;result:sum_b</p>
<p>and again the models output should be two arg:</p>
<ol>
<li>the q’ result (sum_a)</li>
<li>Dot product of sum_a and sum_b</li>
</ol>
<p>This was an another BILSTM explanation,<br>
can you help about this two in torch?</p>

ptrblck: <p>Thanks for the description.<br>
Based on it, it seems you are using the first approach with two LSTMs in Keras, while the second one with the bidirectional LSTMs in PyTorch.<br>
If that’s the case, then surely these architectures are not the same. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>I think I’m not familiar enough with Keras and don’t see how these two LSTM layers can be interpreted as a bidirectional LSTM, but maybe Keras applies something under the hood, which I’m missing.</p>