Questions forthe attention mechanism

username: I_H_Yoo
cooked: <p>Hi. I’m a little bit struggling to implement attention mechanisms and I got questions during implementing it.</p>
<ol>
<li>
<p>Some implements including <a href="https://github.com/SethHWeidman/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" rel="nofollow noopener">PyTorch tutorial</a> uses the last hidden state of the encoder as the initial hidden state of the decoder in Bahdanau attention. I read the original paper, they don’t mention this. So what is the main reason for using the last hidden state of the encoder as the initial hidden state of the decoder?</p>
</li>
<li>
<p>When LSTM is employed to both encoder and decoder, which do I use between hidden state and cell state for calculating the attention score?</p>
</li>
<li>
<p>In Luong attention, the output(=$H$) of the encoder’s dimension is <code>[Seq_len, Batch, num_dir(=2) * hidden_dim]</code>. However, the dimension of the encoder’s output <code>[Seq_len, Batch, num_dir(=2) * enc_hidden_dim]</code> doesn’t match to the decoder’s output <code>[Seq_len, Batch, num_dir(=1) * dec_hidden_dim]</code> when the method of alignment function is dot. So how do I do to handle it?</p>
</li>
</ol>

username: harsha_g
cooked: <aside class="quote no-group" data-username="I_H_Yoo" data-post="1" data-topic="86574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_h_yoo/48/19022_2.png" class="avatar"> I_H_Yoo:</div>
<blockquote>
<p>Some implements including <a href="https://github.com/SethHWeidman/pytorch-seq2seq/blob/master/3%20-%20Neural%20Machine%20Translation%20by%20Jointly%20Learning%20to%20Align%20and%20Translate.ipynb" rel="noopener nofollow ugc">PyTorch tutorial </a> uses the last hidden state of the encoder as the initial hidden state of the decoder in Bahdanau attention. I read the original paper, they don’t mention this. So what is the main reason for using the last hidden state of the encoder as the initial hidden state of the decoder?</p>
</blockquote>
</aside>
<p>The answer to this is in the appendix section A.2.2 of the paper.</p>
<aside class="quote no-group" data-username="I_H_Yoo" data-post="1" data-topic="86574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_h_yoo/48/19022_2.png" class="avatar"> I_H_Yoo:</div>
<blockquote>
<p>When LSTM is employed to both encoder and decoder, which do I use between hidden state and cell state for calculating the attention score?</p>
</blockquote>
</aside>
<p>Hidden</p>
<aside class="quote no-group" data-username="I_H_Yoo" data-post="1" data-topic="86574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_h_yoo/48/19022_2.png" class="avatar"> I_H_Yoo:</div>
<blockquote>
<p>In Luong attention, the output(=$H$) of the encoder’s dimension is <code>[Seq_len, Batch, num_dir(=2) * hidden_dim]</code> . However, the dimension of the encoder’s output <code>[Seq_len, Batch, num_dir(=2) * enc_hidden_dim]</code> doesn’t match to the decoder’s output <code>[Seq_len, Batch, num_dir(=1) * dec_hidden_dim]</code> when the method of alignment function is dot. So how do I do to handle it?</p>
</blockquote>
</aside>
<p>Hopefully, someone will be able to shed more light on this. However, one crude way to make the dot product work is set <code>dec_hidden_dim = 2*enc_hidden_dim</code>.</p>

username: I_H_Yoo
cooked: <p>Thank you very much. The answer is very clear. <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html?highlight=chatbot" rel="nofollow noopener">The pytorch tutorial for chat bot</a> imploies Luong attention and it solves the unmatching dimension of dot align function problem by using summation on hidden dimension.</p>

username: harsha_g
cooked: <p>Glad you figured it out. Good luck with your NLP work. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>

username: I_H_Yoo
cooked: <p>Thanks. However, someone may give me more clearer answer , so I let this topic as ‘unsolved’ for a while.</p>