piskvorky/gensim — pull request #206 by gojomo (2014-06-04 01:10:08+00:00)
Title: 'sample' parameter, for frequent-word down-sampling

Implements 'sample' parameter for down-sampling of frequent words. Interpretation of parameter follows original C word2vec formula. (Word frequencies actually need to be about 2.6x the parameter before their odds-of-inclusion fall below 1.0.)

When down-sampling is enabled, the 'total_words' value used for progress-estimation and alpha-updating is an estimate. (So, final progress update my indicate a value over 100%.) Sampling-probabilities for frequent words are precalculated and stored in the Vocab objects to avoid constant recalculation during training.

Un-sampled words are removed from the sentences, rather than replaced by `None`. Additionally, the handling of out-of-vocabulary words has been changed to remove such words, rather than replace them with `None`. In both cases, this matches my understanding of what the original C word2vec does, and a series of short trials using the 'text8' corpus shows slight accuracy improvements against the question-words.txt accuracy check.