How nn.Embedding trained?

username: juhyung
cooked: <p>when i have vocab size of 40000 and want to embed this to 300</p>
<p>I use nn.Embedding(40000, 300)</p>
<p>then How is embeddings are trained?? Since it is not kind of word2vec task, has no label for each word.</p>

username: se7en
cooked: <p>Embedding is not for training, it’s a lookup table. You first map each word in the vocabulary to a unique integer index, and then the nn.Embedding just map this index to a vector with size of 300.</p>

username: juhyung
cooked: <p>if 300 vector is not trained, that word vector does not represent relations among words. right??</p>

username: se7en
cooked: <p><a href="https://stackoverflow.com/questions/44881999/word-embedding-lookuptable-word-embedding-visualizations" rel="nofollow noopener">https://stackoverflow.com/questions/44881999/word-embedding-lookuptable-word-embedding-visualizations</a></p>
<p>I found this post very helpful. I think the nn.Embedding just initialize the lookup table, and thereafter you train it with gradient descent.</p>

username: ptrblck
cooked: <p><code>nn.Embedding</code> acts like a trainable lookup table.<br>
The relations between words will be learned during its training.<br>
<a href="https://towardsdatascience.com/deep-learning-4-embedding-layers-f9a02d55ac12" rel="nofollow noopener">This blog post</a> might be useful to get some intuition on this layer.</p>

username: Adamits
cooked: <p>The 40,000 word vectors are learned as another parameter of the network that you train. There is a lot of literature on pretraining word embeddings using LSA or the W2V algorithms, but by initializing random vectors (in this size of dimension 300), and applying updates to those with backpropogation, we can learn good approximations of such vectors that are tuned to the objective of your NN.</p>

username: maqboolurrahim
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. But I have another question how does Pytorch learn these embeddings? does it use word2vec Skipgram or CBOW kind of model? I mean can you kindly provide more details?<br>
thanks</p>

username: ptrblck
cooked: <p>An embedding layer is a simple lookup table accepting a sparse input (word index) which will be mapped to a dense representation (feature tensor). The embedding weight matrix will get gradients and will thus be updated. SkipGram etc. would refer to a training technique and your model might use embedding layers for it.</p>

username: HumanIearning
cooked: <p>than, you mean default training technique of nn.embedding is SkipGram?</p>

username: ptrblck
cooked: <p>No, I don’t think <code>SkipGram</code> is the default training technique and my previous post tried to separate the actual layer (nn.Embedding) vs. any learning technique (e.g. <code>SkipGram</code>).</p>

username: dreidizzle
cooked: <p>I think the upshot is nn.Embedding can be viewed as a map from token_id → vector. How you get this vector is up to you. What the token is us up to you also.</p>
<ol>
<li>Skip gram negative sampling: <a href="https://arxiv.org/pdf/1310.4546.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1310.4546.pdf</a><br>
Note here you have technically two embedding tables (context and output). For each token, you can say it’s embedding is either the context embedding or the average of the context and output embeddings. For each word, predict the surrounding words.</li>
<li>CBOW<br>
Same as above. Different model architecture: for each word, the surrounding words predict the word.</li>
<li>Fasttext: <a href="https://fasttext.cc/" rel="noopener nofollow ugc">https://fasttext.cc/</a><br>
Here the tokens are not words but ngrams of chars. For a word, you get the embedding by averaging the embeddings over all continuous ngrams in the word.</li>
<li>Glove: <a href="https://nlp.stanford.edu/projects/glove/" class="inline-onebox" rel="noopener nofollow ugc">GloVe: Global Vectors for Word Representation</a><br>
More of a global objective than the above. Different notion of similarity between words. But, the bottom line is you still have words as tokens and each word gets an embedding.</li>
</ol>
<p>All of the above embeddings are such that a unique token has a unique (universal) vector for the model. There is no notion on “context”. Transformers get token representations with-context. So, in this case you feed in “the river bank” and “I robbed a bank” and you get different vector representations for the token “river” (assuming words are the tokens here). For SkipGram, bank has just one embedding, which you learn via optimization.</p>
<p>Basically Embedding is a building block - your model and configuration decides what it contains and how you train it.</p>

username: vdw
cooked: <aside class="quote no-group" data-username="HumanIearning" data-post="9" data-topic="32533" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/humaniearning/48/56661_2.png" class="avatar"> HumanIearning:</div>
<blockquote>
<p>than, you mean default training technique of nn.embedding is SkipGram?</p>
</blockquote>
</aside>
<p>SkipGram is not a training technique. SkipGram or CBOW are rather network architectures designed to learn meaningful word vectors. For example, SkipGram is set up to predict the surrounding words given a target word (CBOW is vice versa). After the SkipGram model is train using a large corpus, one can extract “some weights” from the model that represent word vectors.</p>
<p>This word vectors can be downloaded and you can use them as initial weights for you <code>nn.Embedding</code> layer. Still, they are just some weights. So if you don’t freeze the embedding layer, they get updated during backpropagation like any other parts of your network.</p>

username: mo_hd
cooked: <p>But what if my vocabulary is very large, yet I haven’t obtained the indices of all my words.<br>
Does nn.Embedding accept indices out of the range of what was given at the initialization?<br>
I cannot load my vocabulary all at once, which means i even don’t know its size yet, i am batching through texts and there could be new words always. How can i fit that layer with those words dynamically?</p>

username: J_Johnson
cooked: <p>Here is a definition you can make use of and some tests showing it works:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

def expand_emb(emb_layer, num_new_emb:int):
    with torch.no_grad():
        new_weights = torch.randn((num_new_emb, emb_layer.embedding_dim))
        emb_layer.weight.data = torch.cat([emb_layer.weight.data,new_weights])
        emb_layer.num_embeddings+=num_new_emb
    return emb_layer

model = nn.Embedding(10000, 300)

print(model.weight.data.size())

dummy_inputs = torch.randint(15000,(12, 1200))

# test model, note int size is larger than allowed so we expect an error
try:
    print(model(dummy_inputs).size())
except Exception as e:
    print(e)

#add new embeddings
model = expand_emb(model, 5000)
#check new size
print(model.weight.data.size())

# test model again
try:
    print(model(dummy_inputs).size())
    print("Model successful")
except Exception as e:
    print(e)
</code></pre>
<p>On a side note, when you add new embeddings, during a training regimen, take note that your learning rate may be lower than when the initial embeddings were trained, and thus the new embeddings may not perform as well as the original embeddings.</p>