How to set dimension problems

username: Gorgen
cooked: <p>Dear everyone:</p>
<p>Sorry, I am new to pytorch.  When I learn tutorial (TRANSLATION WITH A SEQUENCE TO SEQUENCE NETWORK AND ATTENTION).<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2dd63ed7b27be15e6be8c3b49af9b16262f53cb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a2dd63ed7b27be15e6be8c3b49af9b16262f53cb" title="ATJI@3DDK)KL)@JQJ{$Q9}UTPJRAO{ZB5"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2dd63ed7b27be15e6be8c3b49af9b16262f53cb_2_690x365.jpeg" alt="ATJI@3DDK)KL)@JQJ{$Q9}UTPJRAO{ZB5" data-base62-sha1="neLFXKrUKqi6CmcLufi7kEQieyD" width="690" height="365" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2dd63ed7b27be15e6be8c3b49af9b16262f53cb_2_690x365.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2dd63ed7b27be15e6be8c3b49af9b16262f53cb.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2dd63ed7b27be15e6be8c3b49af9b16262f53cb.jpeg 2x" data-dominant-color="EAE5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ATJI@3DDK)KL)@JQJ{$Q9}UTPJRAO{ZB5</span><span class="informations">782×414 49.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>1st question: May I ask why you choose embedded[0] in self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1), and what do hidden[0] represent? Why not take embedded, hidden overall<br>
2nd question:  attn_applied = torch.bmm(attn_weights.unsqueeze(0),<br>
encoder_outputs.unsqueeze(0))<br>
Why is it multiplied after expansion in attn_weights.unsqueeze(0) dimension?</p>
<p>Thanks<br>
best wishes</p>

username: vdw
cooked: <p>At least with respect to <code>hidden</code>, note that the output shape is <code>(num_direction*num_layers, batch_size, hidden_size)</code>. Since the <code>nn.GRU</code> is unidirectional (<code>num_directions=1</code>) and has only one layer (<code>num_layers=1</code>), the shape simplifies to <code>(1, batch_size, hidden_size)</code>. If I remember correctly, this tutorial uses only batch size of 1, so the shape is <code>(1, 1, hidden_size)</code></p>
<p>This means that <code>hidden[0]</code> gives you the last hidden state. I would prefer <code>hidden[-1]</code>. In this case you could increase <code>num_layers</code> without changing the code.</p>
<p>The output shape of <code>self.embedding</code> should be <code>(batch_size,seq_len, embed_dim)</code>. This after <code>.view(1, 1, -1)</code> the shape is <code>(1, 1, batch_size*seq_len*embed_dim)</code>. If I remember correctly, this tutorial uses only batch size of 1, so the shape is <code>(1, 1, seq_len*embed_dim)</code>. Also, this is the decoder where we generate words step by step, so the sequence length is also one, resulting in a shape of <code>(1, 1, embed_dim)</code></p>
<p>Summing up, the shapes are:</p>
<ul>
<li>
<code>embedded[0]</code> → <code>(1, 1, embed_dim)</code>
</li>
<li>
<code>hidden[0]</code> → <code>(1, 1, hidden_size)</code>
</li>
</ul>
<p>In general, this tutorial implements the <a href="https://machinelearningmastery.com/the-bahdanau-attention-mechanism/" rel="noopener nofollow ugc">Bahdanau Attention</a>, which is a bit more confusing to me. I implemented the simpler <a href="https://machinelearningmastery.com/the-luong-attention-mechanism/" rel="noopener nofollow ugc">Luong Attention</a>; <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="noopener nofollow ugc">see here</a>.</p>