Why does the tensor dimension not seem to matter here?

Dusty: <p>Hello everybody,</p>
<p>in this (<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training" rel="noopener nofollow ugc">NMT Tutorial</a>) Tutorial (Section: “Training the Model”), the initial <code>decoder_input</code> is defined like this:</p>
<blockquote>
<p>decoder_input = torch.tensor([[SOS_token]], device=device)</p>
</blockquote>
<p>This returns a tensor that looks like this:</p>
<blockquote>
<p>tensor([[0]], device=‘cuda:0’) torch.Size([1, 1])</p>
</blockquote>
<hr>
<p>Now, after doing a forward pass in the Decoder a few lines below, the next <code>decoder_input</code> is defined like this:</p>
<blockquote>
<p>decoder_input = target_tensor[di]</p>
</blockquote>
<p>This returns a tensor that looks like this:</p>
<blockquote>
<p>tensor([129], device=‘cuda:0’) torch.Size([1])</p>
</blockquote>
<hr>
<p>For context, the decoder input is fed into the forward() function as <code>inp</code>:</p>
<pre><code class="lang-auto">    def forward(self, inp, hidden, encoder_outputs):
        embedded = self.embedding(inp).view(1, 1, -1)
        ...
</code></pre>
<p>The embedding layer is defined as this:</p>
<p><code>self.embedding = nn.Embedding(self.output_size, self.hidden_size)</code> (Output size is vocab size of the input language).</p>
<p><em>Why does the difference in shape not matter here? If it does, what should I use and why?</em></p>

srishti-git1110: <p>Hi,<br>
In the simplest terms, <code>nn.Embedding(4, 3)</code> will act as a look-up table for <em>3-dimensional</em> vectors corresponding to <em>4 indices</em> 0, 1, 2, 3.</p>
<aside class="quote no-group" data-username="Dusty" data-post="1" data-topic="162991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a183cd/48.png" class="avatar"> Dusty:</div>
<blockquote>
<p>Why does the difference in shape not matter here?</p>
</blockquote>
</aside>
<p>It matters inasmuch as it determines the output shape, like so:</p>
<pre><code class="lang-auto">x = torch.tensor([1, 2, 0, 3])  # a single vector, 1 dimensional
y = torch.tensor([[1, 2, 0, 3]]) # batch_size * sequence length
emb = nn.Embedding(4, 3)
x_emb = emb(x)
y_emb = emb(y)
print(x_emb, y_emb)
</code></pre>
<p>gives:</p>
<pre><code class="lang-auto">(tensor([[ 0.4080,  1.3991,  1.1883],
         [-0.3503,  0.1206,  0.2660],
         [ 0.8378, -0.3656,  1.6117],
         [ 1.5515, -1.0316,  1.6244]], grad_fn=&lt;EmbeddingBackward0&gt;),
 tensor([[[ 0.4080,  1.3991,  1.1883],
          [-0.3503,  0.1206,  0.2660],
          [ 0.8378, -0.3656,  1.6117],
          [ 1.5515, -1.0316,  1.6244]]], grad_fn=&lt;EmbeddingBackward0&gt;))
</code></pre>
<p>Notice the difference in output shapes.</p>
<aside class="quote no-group" data-username="Dusty" data-post="1" data-topic="162991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a183cd/48.png" class="avatar"> Dusty:</div>
<blockquote>
<p>If it does, what should I use and why?</p>
</blockquote>
</aside>
<p>I think using an extra dimension for the batch_size brings clarity. Nevertheless, depends on your use-case.</p>

Dusty: <p>Batch size, that makes sense. I see your point, thanks for your quick reply!<br>
If I understand you right, then this would mean, that <code>y</code> in your example corresponds to a batch size of 1.<br>
And since the output of the embedding layer will be reshaped using (<code>.view(1,1,-1</code>), the final output of the embedding layer (<code>embedd</code>) will always be the same, regardless of the extra dimension for batch size.</p>

srishti-git1110: <aside class="quote no-group" data-username="Dusty" data-post="3" data-topic="162991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a183cd/48.png" class="avatar"> Dusty:</div>
<blockquote>
<p><code>y</code> in your example corresponds to a batch size of 1.</p>
</blockquote>
</aside>
<p>Right.</p>
<aside class="quote no-group" data-username="Dusty" data-post="3" data-topic="162991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a183cd/48.png" class="avatar"> Dusty:</div>
<blockquote>
<p>the final output of the embedding layer (<code>embedd</code>) will always be the same, regardless of the extra dimension for batch size.</p>
</blockquote>
</aside>
<p>Yes, so the output from the embedding layer shall have the shape (1, 1, x), where x depends on the batch_size, sequence length etc.</p>