How to change GRU to LSTM in Chatbot Tutorial

StephKua: <p>Hi guys,</p>
<p>I’m trying to use LSTM in the Chatbot tutorial provided by Pytorch. However, I’m currently facing an error shown in below.</p>
<blockquote>
<p>RuntimeError: Expected hidden[0] size (4, 64, 500), got (2, 64, 500)</p>
</blockquote>
<p>Any help would be much appreciated.</p>
<p>Thank you and have a nice day.</p>

ptrblck: <p>The shape of hidden should not change swapping the LSTM for GRU.<br>
Could you post your code here so that we could have a look?</p>

StephKua: <p>Thank you for your reply.</p>
<blockquote>
<p>self.gru = nn.GRU(hidden_size, hidden_size, n_layers, dropout=(0 if n_layers == 1 else dropout), bidirectional=True)</p>
</blockquote>
<p><code>original.</code></p>
<blockquote>
<p>self.lstm = nn.LSTM(hidden_size, hidden_size, n_layers, dropout=(0 if n_layers == 1 else dropout), bidirectional=True)</p>
</blockquote>
<p><code>swapped to LSTM.</code></p>
<p>The only thing that I change was nn.GRU to nn.LSTM for both EncoderRNN and LuongAttnDecoderRNN.</p>
<p>Original Code Link: <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html</a></p>

ptrblck: <p>Thanks for the clarification.<br>
I guess you are just changing the model without modifying the forward pass.<br>
The error might be thrown, if you forget to pass a <code>state</code> tensor to the LSTM.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/nn.html?highlight=rnn#torch.nn.LSTM">docs</a>.<br>
Since <code>hidden</code> and <code>state</code> are assumed to be passed as a tuple, <code>hidden</code> will be sliced in case you pass it without <code>state</code>, which yields this size mismatch error.</p>

Dean_Sumner: <p>Aside from the forward pass modification mentioned above, I believe it is also necessary to change the part of the code in the training function. I.e, instead of passing the last encoder_hidden, make sure you are also passing the last (encoder_ h_hidden, encoder_c_hidden) to the decoder.</p>

vdw: <p>Can you specify where this error is thrown, i.e., in which line of your code or at least whether in the encoder or decoder. When going through the tutorial, I cannot see why switching from GRU to LSTM should cause any troubles. I usually write my models in such a why that the choice of cell is configurable, and there are not many cases to consider.</p>
<p>The hidden state of both GRU and LSTM has a shape of <code>(num_layers * num_directions, batch, hidden_size)</code> – LSTM has also a cell state with the same shape. Looking at your error, the first dimension (i.e., <code>num_layers * num_directions</code>). Without any more details, I would guess that something with the values for <code>num_layers</code> and/or <code>num_directions</code> might be off.</p>

Dean_Sumner: <p>My recommendations need to be used both in the training function and in the evaluation function when setting the decode initial state.</p>
<p>Example code:</p>
<blockquote>
<pre><code> # Set initial decoder hidden state to the encoder's final hidden state
if encoder.model_type == 'GRU':
    decoder_hidden = encoder_hidden[:decoder.n_layers]
    
elif encoder.model_type == 'LSTM':
    #Get the encoder final h_hidden_state
    encoder_h_hidden, encoder_c_hidden= encoder_hidden
    decoder_h_hidden = encoder_h_hidden[:decoder.n_layers]
    decoder_c_hidden = encoder_c_hidden[:decoder.n_layers]
    # Recombine the final hidden states as hidden tuple
    decoder_hidden = (decoder_h_hidden, decoder_c_hidden)
</code></pre>
</blockquote>

Sandip_More: <p>Thanks It worked for me .just to add: put below code in train function and evaluate fubction</p>
<h1>Set initial decoder hidden state to the encoder’s final hidden state</h1>
<p><span class="hashtag">#Get</span> the encoder final h_hidden_state<br>
encoder_h_hidden, encoder_c_hidden= encoder_hidden<br>
decoder_h_hidden = encoder_h_hidden[:decoder.n_layers]<br>
decoder_c_hidden = encoder_c_hidden[:decoder.n_layers]</p>
<h1>Recombine the final hidden states as hidden tuple</h1>
<p>decoder_hidden = (decoder_h_hidden, decoder_c_hidden)</p>

rionel_dmello: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> Can you tell me how to add a state vector. The code works as is if I do what <a class="mention" href="/u/dean_sumner">@Dean_Sumner</a>  said but I feel like I am missing something because I am not passing the state vector. What I am trying to ask is “Is it okay to do only what <a class="mention" href="/u/dean_sumner">@Dean_Sumner</a> said or is there something else I have to do too?”. Thank you in advance.</p>

ptrblck: <p>I’m not completely sure which code you are referring to.<br>
In <a href="https://discuss.pytorch.org/t/how-to-change-gru-to-lstm-in-chatbot-tutorial/30417/7">this post</a> <a class="mention" href="/u/dean_sumner">@Dean_Sumner</a> creates the hidden and cell states.</p>

rionel_dmello: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Yeah I tried it and it works. Thank you so much. I was referring to <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Chatbot Tutorial — PyTorch Tutorials 1.7.1 documentation</a></p>