How should I understand the num_embeddings and embedding_dim arguments for nn.Embedding?

seankala: <p>Hello. I’m aware that this question (and many similar ones) have already been asked on this forum and Stack Overflow, but I’m still having trouble grasping how the concept works and wanted to ask a question based on a specific toy example that I went through.</p>
<p>I’m aware that the <code>num_embeddings</code> argument refers to how many elements we have in our vocabulary, and <code>embedding_dim</code> is simply referring to how many dimensions we want to make the embeddings.</p>
<p>The specific code that I tried is as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


embedding = nn.Embedding(num_embeddings=10, embedding_dim=3)

a = torch.LongTensor([[1, 2, 3, 4], [4, 3, 2, 1]]) # (2, 4)

b = torch.LongTensor([[1, 2, 3], [2, 3, 1], [4, 5, 6], [3, 3, 3], [2, 1, 2],
                      [6, 7, 8], [2, 5, 2], [3, 5, 8], [2, 3, 6], [8, 9, 6],
                      [2, 6, 3], [6, 5, 4], [2, 6, 5]]) # (13, 3)

c = torch.LongTensor([[1, 2, 3, 2, 1, 2, 3, 3, 3, 3, 3],
                      [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]]) # (2, 11)
</code></pre>
<p>If I run <code>a</code>, <code>b</code>, and <code>c</code> through <code>embedding</code> then I get embedding tensors each of shape <code>(2, 4, 3)</code>, <code>(13, 3, 3)</code>, <code>(2, 11, 3)</code>.</p>
<p>My question here is, shouldn’t <code>b</code> give me an index out of range error, since it’s a tensor consisting of 13 words each of dimension 3, and hence is outside the range of the predefined 10?</p>
<p>Any tips or pointers are appreciated. Thanks in advance.</p>

vainaijr: <p>I think when you do</p>
<pre><code class="lang-auto">embedding = nn.Embedding(num_embeddings=10, embedding_dim=3)
</code></pre>
<p>then it means that you have 10 words and represent each of those words by an embedding of size 3, for example, if you have words like</p>
<pre><code class="lang-auto">hello
</code></pre>
<pre><code class="lang-auto">world
</code></pre>
<p>and so on, then each of these would be represented by 3 numbers,<br>
one example would be,</p>
<pre><code class="lang-auto">hello -&gt; [0.01 0.2 0.5]
world -&gt; [0.04 0.6 0.7]
</code></pre>
<p>and so on, if you do</p>
<pre><code class="lang-auto">list(embedding.parameters())
</code></pre>
<p>then you will get something like this,</p>
<pre><code class="lang-auto">[Parameter containing:
 tensor([[ 0.9227,  0.6492, -1.1440],
         [ 1.5318, -0.2873, -0.7290],
         [-0.4234, -1.7012, -0.9684],
         [-0.2859,  1.4677, -1.4499],
         [-1.8966, -1.4591,  0.5218],
         [ 2.4023, -1.5395, -0.7947],
         [-0.0464,  0.7174, -0.7452],
         [ 0.9500, -0.4633,  0.5398],
         [ 0.3458, -0.7997,  0.8895],
         [-0.3303, -0.5663, -0.2300]], requires_grad=True)]
</code></pre>
<p>which represents how are each of these words represented,</p>
<p>when you do,</p>
<pre><code class="lang-auto">a = torch.LongTensor([[1, 2, 3, 4], [4, 3, 2, 1]]) # (2, 4)
</code></pre>
<p>and then</p>
<pre><code class="lang-auto">embedding(a).shape
</code></pre>
<p>it gives</p>
<pre><code class="lang-auto">torch.Size([2, 4, 3])
</code></pre>
<p>while</p>
<pre><code class="lang-auto">embedding(a)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">tensor([[[ 1.5318, -0.2873, -0.7290],
         [-0.4234, -1.7012, -0.9684],
         [-0.2859,  1.4677, -1.4499],
         [-1.8966, -1.4591,  0.5218]],

        [[-1.8966, -1.4591,  0.5218],
         [-0.2859,  1.4677, -1.4499],
         [-0.4234, -1.7012, -0.9684],
         [ 1.5318, -0.2873, -0.7290]]], grad_fn=&lt;EmbeddingBackward&gt;)
</code></pre>
<p>because you are retrieving the embeddings of those words, means, you are asking give me the embedding of word at index 1, give me embedding of word at index 2, and so on. So, it gives you embeddings of words at indices that you asked.</p>
<p>when you do</p>
<pre><code class="lang-auto">b = torch.LongTensor([[1, 2, 3], [2, 3, 1], [4, 5, 6], [3, 3, 3], [2, 1, 2],
                      [6, 7, 8], [2, 5, 2], [3, 5, 8], [2, 3, 6], [8, 9, 6],
                      [2, 6, 3], [6, 5, 4], [2, 6, 5]]) # (13, 3)
embedding(b)
</code></pre>
<p>then it means gives me the embedding of word at index 1, give me embedding of word at index 2, then 3, then 2, then 3, then 1 and so on.</p>
<p>here, ‘a’ and ‘b’ contain indices of words you want to retrieve the embedding for.</p>

AntoineC: <p>The <em>values</em> in <code>a</code> must all be between <code>1</code> and  <code>num_embeddings</code>:</p>
<pre><code class="lang-python">for i in range(num_embeddings + 4):
    try:
        embedding(torch.arange(i))
    except:
        print(f"failed for i={i}")
</code></pre>
<p>The input data can be any shape:</p>
<pre><code class="lang-python">try:
    embedding(torch.randint(1, num_embeddings, size=(2,3,4,5,6)))
    print("it worked")
except:
    print("this won't print because this won't fail")
</code></pre>
<p>Each entry in your input tensor is mapped to a vector with 3 coordinates (a 3-dimensional vector in mathematical terminology, but <em>not</em> in the sense of <code>PyTorch</code> <code>tensor</code>s), which can be found in the last axis of the output tensor.  Namely, <code>a[i][j]</code> is mapped to the vector <code>embedding(a)[i][j]</code>, which is a <code>0</code>-dimensional <code>tensor</code> with 3 components.</p>
<p><strong>Interpretation of dimensions</strong><br>
Say for example your input <code>tensor</code> has <code>shape</code> <code>(13, 3)</code>, with values between <code>1</code> and <code>10</code>.   If this were to represent text, then you can think of it as having <code>13</code> samples of text each containing <code>3</code> words each, and each word is taken from a vocabulary of <code>10</code> words.</p>

AbdulsalamBande: <p>Hello, Do not forget that the <code>13</code> is just a dimension. What really matters is the indexes in the b vector. Your number of embedding is <code>10</code>. So the values of your input all have to be <code>less than or equal to 10</code> and you have satisfied that condition in the <code>b tensor</code>. All of the <code>13 by 3 tensors</code> have be projected to a <code>13 by 3 by 3</code> space. Thank you</p>

seyeeet: <p>what is the use of embedding, how people use it ?</p>

Unity05: <p>Hello vainaijr,</p>
<p>I agree with you. I would add that PyTorch’s Tutorial specifically on Word Embeddings does a good job with communicating an intuition (<a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html</a>).</p>
<p>As you have kind of focused on the transformer architecture and also written about a use in CV, I just want to throw in this <em>relatively</em> new paper which could be interesting (<a href="https://openreview.net/forum?id=YicbFdNTTy" rel="noopener nofollow ugc">https://openreview.net/forum?id=YicbFdNTTy</a>).</p>
<p>Furthermore, I don’t think these embeddings (speaking about word embeddings) claim to consider order, that’s why we have positional encoding in transformers for instance.<br>
I guess one of the main advantages in using (word) embeddings is that we have dense vectors and also the ability to ‘compare the meanings’ of words just using the embeddings.</p>
<p>Greetings,<br>
Unity05</p>

seyeeet: <p>thank you so much for the explanation it clears out a lot of fogs for me.<br>
do you by any chance know a simple example (beyond the pytorch tutorial) that I can look into for understanding even better?</p>

Andre_Godinho: <p>For Question Answering tasks, can we use nn.Embedding to represent role embeddings? In other words, the agent role embedding and the user role embedding are both trainable. The reasoning behind this is to use some embedding representations for agent and user utterances (GloVe, fastText, nn.Embedding, BERT embeddings, etc.), and we could add these trainable embeddings to the utterance representation according to the role utterance to help the model distinguish agent and user utterances.</p>
<p>For example, the authors of <a href="https://arxiv.org/abs/1911.06192" rel="noopener nofollow ugc">Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering</a> apply this idea (search for <em>role embedding</em> on the .pdf). I would like to do something similar with torch.</p>

Unity05: <p>Hi,</p>
<p>spontaneously, I don’t see a reason, why it should not work. I haven’t known this paper before and I’m curious if you’ve tried it. If yes, did it go well? ^^</p>