Problems about official tutorial ‘Translation with a Sequence to Sequence Network and Attention’

username: plus_real
cooked: <p>I am learning attention mechanism through reading official tutorial ‘Translation with a Sequence to Sequence Network and Attention’<br>
but I think there is something wrong about implementation of attention decoder:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fb4a779beba8c97c69f765288946759eed8f2789.png" data-download-href="https://discuss.pytorch.org/uploads/default/fb4a779beba8c97c69f765288946759eed8f2789" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/fb4a779beba8c97c69f765288946759eed8f2789.png" alt="image" data-base62-sha1="zR1ocAzwOdpdpaKgXJSI8SygCp3" width="593" height="500" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">776×654 69.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>we can find that attention weights are calculated by code below：<br>
<code>attn_weights = F.softmax(self.attn(torch.cat((embedded[0], hidden[0]), 1)))</code></p>
<p>however，the dimention of hidden is <code>(num_layers * num_directions, batch, hidden_size)</code><br>
while the dimention of embedded is  <code>(batch, seq_len, embedding_dim)</code> or <code>(seq_len, batch, embedding_dim)</code><br>
Obviously <code>seq_len</code> is usually not equal with  <code>num_layers * num_directions</code> ， so it is  reasonable to concatenate hidden and embedded？</p>
<p>I am a beginner of NLP and hope that someone who understand the tutorial can explain the reason why they concatenate hidden and embedded, Thank you!</p>

username: siddharth_sharma
cooked: <p>I am late but for other beginners like myself I’ll try to answer this.<br>
Let’s see each component individually:</p>
<ol>
<li>Embedding matrix: The dimension of <code>embedded</code> is <code>(seq_len, batch, embedding_dim)</code>. Important thing to note in this tutorial is that we are passing input one at a time to network i.e input will be “he” then “is” then “painting” so on rather than “he is painting” in one go. This means <code>seq_len</code> will always be <code>1</code>. So <code>embedded</code> will be of shape <code>(1, batch, embedding_dim)</code> and hence <code>embedded[0]</code> will be of shape <code>(batch, embedding_dim)</code>.</li>
<li>Hidden weights: the dimension of hidden is  <code>(num_layers * num_directions, batch, hidden_size)</code>. For this tutorial <code>num_layers=1</code> and <code>num_directions=1</code>  so the dimension becomes <code>(1 * 1, batch, hidden_size)</code>. So <code>hidden[0]</code> becomes <code>(batch, hidden_size)</code>
</li>
</ol>
<p>From above to if we concatenate embedded[0] (batch, embedding_dim) and hidden[0] <code>(batch, hidden_size)</code> along <code>dim=1</code> we safely get tensor of size <code>(batch, hidden_size + embedding_dim)</code> for each time step of the input.</p>
<p>Hopefully it Helps!</p>