Question about Embeddings

username: setsailforfail
cooked: <p>I would like to build a Transformer from scratch, that predicts a certain number of a sequence.<br>
However, I lack some understanding of how embeddings work. I understand the basic concept, but my current understanding seems not to be true in practice:</p>
<p>Assume I have a tensor of the following shape:</p>
<p><code>torch.Size([8000, 4])</code>,</p>
<p>where each element in the first dimension is an integer between 0 and 9:</p>
<pre><code class="lang-auto">tensor([[9, 9, 7, 8],
        [2, 4, 1, 6],
        [9, 7, 1, 0],
        ...,
        [8, 7, 1, 4],
        [4, 2, 8, 0],
        [9, 1, 4, 7]])
</code></pre>
<p>I now want to apply an embedding on that tensor:</p>
<pre><code class="lang-auto">embedding_layer = nn.Embedding(10, 500)
transform = embedding_layer(data)
</code></pre>
<p>As expected, the returned shape of transform is <code>torch.Size([8000, 4, 500])</code>, since ever element has been transformed into the embedded_dimension 500. But I also expect a dictionary size of 10, but where can I find it? If it exists, I expect some attribute of the layer showing me 10, bluntly said.<br>
Can someone  tell me, if my understanding is correct? And if so, can PyTorch show the size of the vocab?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="setsailforfail" data-post="1" data-topic="169857">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/setsailforfail/48/30110_2.png" class="avatar"> setsailforfail:</div>
<blockquote>
<p>But I also expect a dictionary size of 10, but where can I find it?</p>
</blockquote>
</aside>
<p>You can find the <code>num_embeddings=10</code> in the shape of the <code>weight</code> which acts as a lookup table:</p>
<pre data-code-wrap="python"><code class="lang-python">emb = nn.Embedding(num_embeddings=10, embedding_dim=500)
print(emb.weight.shape)
# torch.Size([10, 500])
</code></pre>

username: akuysal
cooked: <p>The following code works.</p>
<p>import torch</p>
<p>input = torch.randint(1,10, [8000, 4])<br>
embedding_layer = torch.nn.Embedding(10, 500)<br>
transform = embedding_layer(input)<br>
print(“Final output”)</p>
<p>However, we will get an error if we change 10 to 11 in the next part. Size of the dictionary of embeddings (num_embeddings) will be index out of range. You can not see num_embeddings as a dimension in transform variable but it has an effect in the boundaries of input.</p>
<p>import torch</p>
<p>input = torch.randint(1,11, [8000, 4])<br>
embedding_layer = torch.nn.Embedding(10, 500)<br>
transform = embedding_layer(input)<br>
print(“Final output”)</p>
<p>I hope it may help you <img src="https://discuss.pytorch.org/images/emoji/apple/raised_hand.png?v=12" title=":raised_hand:" class="emoji" alt=":raised_hand:" loading="lazy" width="20" height="20"></p>

username: J_Johnson
cooked: <p>The embedding layer is learnable. So you can add additional rows on dim=0 to the weights if adding new vocab.</p>
<pre><code class="lang-auto">with torch.no_grad():
    old_weights=model.embeddings.weights
    new_weights=torch.cat([old_weights, torch.rand(1, old_weights.size(1))/old_weights.size(1)])
    model.embeddings.weights=new_weights

</code></pre>