[seq2seq] Initial hidden state of decoder

username: Introvert_Loner
cooked: <p>Hi there!<br>
I’m reading the <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="noopener nofollow ugc">Chatbot Tutorial</a>, and encounter this line of code in the training function that confuses me:<br>
<code># Set initial decoder hidden state to the encoder's final hidden state     decoder_hidden = encoder_hidden[:decoder.n_layers]</code><br>
As far as I know (and tested), the hidden states of Pytorch Bidirectional RNNs (vanilla RNN, GRU, LSTM) contains forward and backward directions alternatively (e.g. we have [4 layers*2 directions, N,h], then the hidden states of forward directions are stored in [0],[2],[4].[6] and backward directions are [1],[3],[5],[7]). <strong>AND</strong> the backward hidden states are stored only the first sequence (initial timestep)<br>
However, the line of code above passes [0:3] (if n_layers of decoder = 4), meaning that it passes the backward hidden states of the initial timestep (of the 1st and 2nd layer) into the decoder, along with the forward hidden states of the 1st and 2nd layer<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10ca1e890fbdb3133582f18570811f9b9699717e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/10ca1e890fbdb3133582f18570811f9b9699717e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10ca1e890fbdb3133582f18570811f9b9699717e_2_690x357.jpeg" alt="image" data-base62-sha1="2owG2yEFHscmJfOtnoILbs0f6cS" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10ca1e890fbdb3133582f18570811f9b9699717e_2_690x357.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10ca1e890fbdb3133582f18570811f9b9699717e_2_1035x535.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10ca1e890fbdb3133582f18570811f9b9699717e_2_1380x714.jpeg 2x" data-dominant-color="F7F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1865×966 169 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<strong>QUESTION</strong>:<br>
Shoundn’t only the forward hidden states (at timestep Tx) be passed into the decoder, or be the mean or sum between the forward and backward directions rather than passing like the line of code?</p>
<p>Thank you in advance!</p>

username: vdw
cooked: <p>Hm, I can see why the code would run, but I don’t like it.</p>
<p>For one, shouldn’t it be <code>encoder_hidden[:-decoder.n_layers]</code> (i.e., with the minus sign)? For example, if the encoder has 5 layers and the decoder just one, <code>encoder_hidden[:1]</code> would be referring to the first layer of the decoder.</p>
<p>Also the encoder can be bidirectional (default), but the decoder must be unidirectional. Do for example, if the encoder is bidirectional but with only 1 layer, and the decoder has 2 layers, the 1st encoder layer will be initialized with the last hidden state of the forward pass, and the 2nd decoder layer will be initialized with the last hidden state of the backward pass. I’m mean, sure, it works, it just doesn’t seem intuitive to me.</p>
<p>So either I’m missing something – more than likely – or I just don’t like that snippet of the implementation :).</p>

username: Introvert_Loner
cooked: <p>I agree with passing hidden states of last layers, but I still am with my idea</p>
<blockquote>
<p>Shoundn’t only the forward hidden states (at timestep Tx) be passed into the decoder, or be the mean or sum between the forward and backward directions rather than passing like the line of code <code>encoder_hidden[:decoder.n_layers]</code>?</p>
</blockquote>
<p>Does my intuition make more sense?</p>

username: vdw
cooked: <p>The main purpose of the decoder to generate/encoder the input sentence into a latent representation, which is often the just the last hidden state of the RNN/LSTM/GRU layer.</p>
<p>How this latent representation is generated, is actually not that important – meaning, you could even use only the backward pass of the input sentence. As such, there’s no principle reason to use only the last hidden state of the backward pass as the initial state of the decoder.</p>
<p>Yes, personally I would go with the mean or some of the last hidden states of the forward and backward pass.</p>
<p>As soon as the decoder has multiple layers, there additional approaches. For example, you could take only the last hidden state of the forward/backward/both pass of the <strong>final</strong> layer, and the repeat it <code>decoder.n_layers</code> time to serve as initial hidden state of the decoder.</p>

username: Introvert_Loner
cooked: <p>Thank you, Chris! This is really intuitive!</p>

username: vdw
cooked: <p>These are good and important questions that I feel get often a bit neglected, at least when I sometimes look into some Github repositories. It’s too convenient to be happy with solution that doesn’t throw an error and kind of works (i.e., it’s training seemingly well).</p>
<p>I’m pretty sure that many adopt the Chatbot tutorial “as is” without trying to dig deep another to understand each step. Again, there’s the chance I’m missing something, but I think to current code – i.e., the line you’ve pointed out – is suboptimal or even wrong.</p>
<p>Well, yes, I would argue that it is wrong, as it does not matter how many layers the encoder have since <code>encoder_hidden[:decoder.n_layers]</code> will always take the “lowest” layers.</p>