How to implement skip-gram or CBOW in pytorch

WMN7: <p>I just learn about word embedding and I think the word vector can be learned by CBOW or Skip-gram procedure. And I have two questions about word embedding in Pytorch.</p>
<h3>The first one–How to understand nn.Embedding in Pytorch</h3>
<p>I think I don’t have a good understanding of Embedding in Pytorch. Is the nn.Embedding has the same function with nn.Linear in Pytorch. I think the nn.Embedding just like shallow fully connected network.</p>
<p>And, if not, how the weights of nn.Embedding are fine-tuned during the training process.</p>
<hr>
<h3>The second one–How to implement skip-gram(or CBOW) in Pytorch</h3>
<p>The second question, I want to to know how to implement skip-gram(or CBOW) in Pytorch, are the following two networks correctly implement CBOW and Skip-gram. (The weight of nn.Embedding is the word vector)</p>
<pre><code class="lang-auto">class CBOWModeler(nn.Module):
    def __init__(self, vocab_size, embedding_dim, context_size):
        super(CBOWModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size*embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)
        
    def forward(self, x):
        embeds = self.embeddings(x).view(1,-1)
        output = self.linear1(embeds)
        output = F.relu(output)
        output = self.linear2(output)
        log_probs = F.log_softmax(output, dim=1)
        return log_probs
</code></pre>
<pre><code class="lang-auto">class SkipgramModeler(nn.Module):
    def __init__(self, vocab_size, embedding_dim, context_size):
        super(SkipgramModeler, self).__init__()
        self.context_size = context_size
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(embedding_dim, 128)
        self.linear2 = nn.Linear(128, context_size*vocab_size)
    
    def forward(self,x):
        embeds = self.embeddings(x).view(1,-1)
        output = self.linear1(embeds)
        output = F.relu(output)
        output = self.linear2(output)
        log_probs = F.log_softmax(output, dim=1).view(self.context_size, -1)
        return log_probs
</code></pre>
<p>So, if I want to use embedding in nlp task, <strong>should I first train like above to obtain to weight of embedding.</strong></p>
<p><strong>Thanks for replying.</strong></p>

n0obcoder: <ol>
<li>
<p>Treat  <code>nn.Embedding</code>  as a lookup table where the key is the word index and the value is the corresponding word vector. However, before using it you should specify the size of the lookup table, and initialize the word vectors.</p>
<p>Not all the weights are trained at the same time in this nn.Embedding. Weight training would depend on your training pairs.  For example, lets say (‘Bruce’ , ‘Wayne’) is a training pair. Assuming that ‘Bruce’ and ‘Wayne’ words are present in your vocabulary with indices 100 and 200(just an example), nn.Embedding would allow you to pick the untrained word vectors for these two indices. This these two vectors would be brought closer to each other, resulting in their training.</p>
<p>Remember nn.Embedding is a lookup table. You just need to give in the indices of the words , and it gives you the word vectors for those words.</p>
</li>
<li>
<p>You can have a look at this pytorch implementation of  <a href="https://github.com/n0obcoder/skip-gram-model" rel="nofollow noopener">Skip-Gram model</a></p>
</li>
</ol>
<p>I hope it helped you !!! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

WMN7: <p>Thanks for your replying.</p>
<p>And about the first question, I still have some questions. As you said below:</p>
<blockquote>
<p>For example, lets say (‘Bruce’ , ‘Wayne’) is a training pair. Assuming that ‘Bruce’ and ‘Wayne’ words are present in your vocabulary with indices 100 and 200(just an example), nn.Embedding would allow you to pick the untrained word vectors for these two indices. This these two vectors would be brought closer to each other, resulting in their training.</p>
</blockquote>
<p>How can these two vectors bring closer to each other. The following are some of my thoughts, and I don’t know whether it is right.</p>
<hr>
<p>I want to make this problem easy to explain, so just assume there are only two words in all, so the word2index is <code>{'Bruce':0, '‘Wayne’:1}</code>. And we assume the initial weight of <code>nn.Embedding</code> is [[0.1,0.2,0.3],[0.2,0.5,0.6]](This is the random number).</p>
<p>When we input word ‘Wayne’(the indice 1) to the <code>nn.Embedding</code>, is it equal to Matrix multiplication, as shown in the picture below.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/547c0bee8f18a017798fe5280ef21f2f5500a770.jpeg" alt="Snipaste_2019-06-12_17-45-31" data-base62-sha1="c3nSjVratIm10LaX2YQoU8vPeso" width="510" height="50"></p>
<p>In this way, the network can do back-propagated to modify the weights of <code>nn.Embedding</code>. So, I think it is very similar to <code>nn.Linear</code>.</p>

n0obcoder: <p>no, you dont need to do matrix multiplication to get the word embedding for the word ‘Wayne’. All you need to do is pass the index of ‘Wayne’ (index = 1) and the nn.Embedding gets your the row =1.</p>
<p>nn.Linear would do matrix multiplication and the result would have been same. But that would be computationally expensive. This is where nn.Embeddings come into picture. All they need is the indices of the words, and they return the word vectors of those words. Its just a lookup table. Less computation is required in this case.</p>

WMN7: <p>Maybe I don’t make this question clear.</p>
<p>I know we can just put the index of word to <code>nn.Embedding</code> and then we can get the word vector.</p>
<p>But if the <code>nn.Embedding</code> is just a lookup table, how can it use BP algorithm to optimize its weight.(If it don’t do matrix multiply.)</p>

dhananjayraut: <p>That’s exactly why you use <code>nn.Embedding</code> .<br>
so that the backpropagation can happen. it is handled in the library.</p>

n0obcoder: <p>in nn.Embeddings back propagation wouldnt happen on the entire matrix. Back propagation will be done only on the rows of the embedding matrix whose indices are passed.</p>

WMN7: <p>Thanks for your patient explanation. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> I can probably understand it now.</p>
<p>I now think <code>nn.Embedding</code> optimizes for embedding(index to vector). We can use <code>nn.Linear</code> to achieve the same result, but if we use <code>nn.Embedding</code>, it can have less computation because b-p wouldn’t happen on the entire matrix.</p>

n0obcoder: <p>its always good  to see a person getting satisfied with your explaination <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Fei_Gao: <p>Hello, thanks for your kind explanation, I now learn the difference between <code>nn.Embedding</code> and <code>nn.Linear</code>，But I cannot find the implementation of BP from the pytorch code, <code>/home/linux/miniconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py</code>.<br>
I find the <code>F.embedding</code> function in <code>/home/linux/miniconda3/lib/python3.7/site-packages/torch/nn/functional.pyi</code>,  which only a function defination:</p>
<pre><code class="lang-python">def embedding(
    input: Tensor, weight: Tensor, padding_idx: Optional[int] = ...,
    max_norm: Optional[float] = ..., norm_type: float = ...,
    scale_grad_by_freq: bool = ..., sparse: bool = ...) -&gt; Tensor: ...
</code></pre>

jmaronas: <p>from what I can see in your blog, I think your implementation of the skip-gram model is not very precise, since starts from one-hot vectors rather than randomized vectors.</p>
<p>In the paper it says: <em>More precisely, we use each current word as an input to a log-linear classifier with continuous projection layer</em>.</p>
<p>So from what I have understood from paper equations basically take random dense embed matrix W and use it as the linear projection to the softmax output, when multiplied by input vector.</p>