Word2vec CBOW mismatch

belerico: <p>Hi everyone, this is my first post here, so sorry if I’m missing something.<br>
I’ve implemented the word2vec algorithm following the <a href="https://github.com/chrisjmccormick/word2vec_commented/blob/master/word2vec.c" rel="nofollow noopener">c code</a> posted by Mikolov, that can be found <a href="https://github.com/belerico/word2vec" rel="nofollow noopener">here</a>.<br>
This is my implementation of the CBOW algorithm with negative sampling, where</p>
<ul>
<li>
<code>u_embs</code> are of dim <code>[N, H]</code>, so <code>N</code> target vectors of dim <code>H</code> to be predicted by the context vectors</li>
<li>
<code>v_embs</code> are of dim <code>[N, C, H]</code>, so for each <code>N</code> target vectors, I get <code>C</code> context vectors of dim <code>H</code>
</li>
<li>
<code>neg_v</code> are of dim <code>[N, M, H]</code>, so for each <code>N</code> target vectors, I get <code>M</code> negative vectors of dim <code>H</code>, where <code>M</code> is the negative sampling size choosen by the user</li>
<li>
<code>pos_u, pos_v, neg_v</code> contains the word ids of the target, context and negative examples respectively. <code>pos_v</code> has been padded with 0 to uniform the size of the context vectors</li>
</ul>
<pre><code class="lang-python">class CBOW(Word2Vec):
    def __init__(self, emb_size, emb_dimension, cbow_mean=True):
        super(CBOW, self).__init__(emb_size, emb_dimension)
        self.cbow_mean = cbow_mean
        init_range = 0.5 / self.emb_dimension
        init.uniform_(self.v_embs.weight.data, -init_range, init_range)
        init.constant_(self.u_embs.weight.data, 0)
        self.v_embs.weight.data[0, :] = 0  # Set padding vector to 0

    def forward(self, pos_u, pos_v, neg_v):
        u_embs = self.u_embs(pos_u)  # u_embs are the "target" vectors
        v_embs = self.v_embs(pos_v)  # v_embs are the "context" vectors

        # Mean of context vector without considering padding idx (0)
        if self.cbow_mean:
            mean_v_embs = torch.true_divide(
                v_embs.sum(dim=1),
                (pos_v != 0).sum(dim=1, keepdim=True),
            )
        else:
            mean_v_embs = v_embs.sum(dim=1)

        score = torch.mul(u_embs, mean_v_embs)
        score = torch.sum(score, dim=1)
        score = F.logsigmoid(score)

        neg_score = torch.bmm(self.v_embs(neg_v), u_embs.unsqueeze(2))
        neg_score = F.logsigmoid(-1 * neg_score)

        return -1 * (score.sum() + neg_score.sum())
</code></pre>
<p>All is working fine, execept for the results that I have when I try to evaluate the learned embeddings (I save the <code>self.v_embs</code> embedding).<br>
Since I’ve also implemented the Skip-Gram algorithm, and the only thing that changes is the line <code>score = torch.mul(u_embs, mean_v_embs)</code>, which becomes <code>score = torch.mul(u_embs, v_embs)</code> (in Skip-Gram <code>u_embs</code> and <code>v_embs</code> has the same dimensions) and there’s no mean to be computed, and since with the Skip-Gram algorithm I obtain similar results to the ones of <code>gensim</code> and Mikolov, I’m wondering if the culprit could be the mean computation.</p>
<p><strong>Gensim CBOW</strong></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Serial</th>
<th>Dataset</th>
<th>Num Pairs</th>
<th>Not found</th>
<th>Rho</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>EN-SimVerb-3500.txt</td>
<td>3500</td>
<td>255</td>
<td>0.1324</td>
</tr>
<tr>
<td>2</td>
<td>EN-YP-130.txt</td>
<td>130</td>
<td>12</td>
<td>0.1754</td>
</tr>
<tr>
<td>3</td>
<td>EN-RG-65.txt</td>
<td>65</td>
<td>0</td>
<td>0.4973</td>
</tr>
<tr>
<td>4</td>
<td>EN-MEN-TR-3k.txt</td>
<td>3000</td>
<td>13</td>
<td>0.5335</td>
</tr>
<tr>
<td>5</td>
<td>EN-WS-353-REL.txt</td>
<td>252</td>
<td>1</td>
<td>0.5849</td>
</tr>
<tr>
<td>6</td>
<td>EN-SIMLEX-999.txt</td>
<td>999</td>
<td>7</td>
<td>0.2567</td>
</tr>
<tr>
<td>7</td>
<td>EN-MTurk-771.txt</td>
<td>771</td>
<td>2</td>
<td>0.5081</td>
</tr>
<tr>
<td>8</td>
<td>EN-MC-30.txt</td>
<td>30</td>
<td>0</td>
<td>0.5343</td>
</tr>
<tr>
<td>9</td>
<td>EN-RW-STANFORD.txt</td>
<td>2034</td>
<td>1083</td>
<td>0.3422</td>
</tr>
<tr>
<td>10</td>
<td>EN-WS-353-ALL.txt</td>
<td>353</td>
<td>2</td>
<td>0.6282</td>
</tr>
<tr>
<td>11</td>
<td>EN-WS-353-SIM.txt</td>
<td>203</td>
<td>1</td>
<td>0.6768</td>
</tr>
<tr>
<td>12</td>
<td>EN-MTurk-287.txt</td>
<td>287</td>
<td>3</td>
<td>0.6159</td>
</tr>
<tr>
<td>13</td>
<td>EN-VERB-143.txt</td>
<td>144</td>
<td>0</td>
<td>0.3538</td>
</tr>
</tbody>
</table>
</div><p><strong>Mine CBOW</strong></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Serial</th>
<th>Dataset</th>
<th>Num Pairs</th>
<th>Not found</th>
<th>Rho</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>EN-SimVerb-3500.txt</td>
<td>3500</td>
<td>255</td>
<td>0.1031</td>
</tr>
<tr>
<td>2</td>
<td>EN-YP-130.txt</td>
<td>130</td>
<td>12</td>
<td>0.1235</td>
</tr>
<tr>
<td>3</td>
<td>EN-RG-65.txt</td>
<td>65</td>
<td>0</td>
<td>0.3562</td>
</tr>
<tr>
<td>4</td>
<td>EN-MEN-TR-3k.txt</td>
<td>3000</td>
<td>13</td>
<td>0.4226</td>
</tr>
<tr>
<td>5</td>
<td>EN-WS-353-REL.txt</td>
<td>252</td>
<td>1</td>
<td>0.4534</td>
</tr>
<tr>
<td>6</td>
<td>EN-SIMLEX-999.txt</td>
<td>999</td>
<td>7</td>
<td>0.2395</td>
</tr>
<tr>
<td>7</td>
<td>EN-MTurk-771.txt</td>
<td>771</td>
<td>2</td>
<td>0.4255</td>
</tr>
<tr>
<td>8</td>
<td>EN-MC-30.txt</td>
<td>30</td>
<td>0</td>
<td>0.5637</td>
</tr>
<tr>
<td>9</td>
<td>EN-RW-STANFORD.txt</td>
<td>2034</td>
<td>1083</td>
<td>0.3147</td>
</tr>
<tr>
<td>10</td>
<td>EN-WS-353-ALL.txt</td>
<td>353</td>
<td>2</td>
<td>0.5190</td>
</tr>
<tr>
<td>11</td>
<td>EN-WS-353-SIM.txt</td>
<td>203</td>
<td>1</td>
<td>0.5775</td>
</tr>
<tr>
<td>12</td>
<td>EN-MTurk-287.txt</td>
<td>287</td>
<td>3</td>
<td>0.5172</td>
</tr>
<tr>
<td>13</td>
<td>EN-VERB-143.txt</td>
<td>144</td>
<td>0</td>
<td>0.3202</td>
</tr>
</tbody>
</table>
</div><p>I know that the results are very similar, but not for all tests and beware that with Skip-Gram the results differ by 1% to 3%.<br>
Sorry for the long post, and thank you all.<br>
Federico</p>

szcCL2: <p>the difference here might caused by implement of CBOW in gensim.<br>
when use sum of context word as input, it is divided by num of context when BP which is contrary to pytorch inner BP algorithm.<br>
check discuss here <a href="https://github.com/RaRe-Technologies/gensim/issues/1873" class="inline-onebox" rel="noopener nofollow ugc">Is this a bug in the CBOW code or my misunderstanding? · Issue #1873 · RaRe-Technologies/gensim · GitHub</a></p>