piskvorky/gensim — pull request #176 by sebastien-j (2014-03-05 01:09:14+00:00)
Title: CBOW

I have added the CBOW functionality.

It should replicate the original code by Mikolov et al., except that it combines context words by taking the average of the corresponding vectors instead of their sum. This is in agreement with the textual description of the CBOW model in [1] (although Figure 1 in that paper says "SUM").

Here are some baseline results on the Google analogy dataset (questions-words.txt) with the vocabulary restricted to the 30,000 most frequent words. The training corpora are the "text8" and "fil9" datasets found at "http://mattmahoney.net/dc/textdata.html". In all cases, I used vectors of dimension 640 and a window of size 10. The rest of the hyper-parameters take their default value.

With "text8":

Skip-gram: 29.3% correct
CBOW: 15.5% (11.5% if I used the sum of the vectors instead of their average)

With "fil9":

Skip-gram: 52.5%
CBOW: 46.2% (29.9% using the sum)

Note: By default, my system always used "Optimization 1" during training. What would be the most efficient way to test the other optimizations before incorporating the changes into Gensim?

-------------
[1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.