piskvorky/gensim — pull request #170 by josephcc (2014-02-17 21:19:49+00:00)
Title: Filter wikipedia articles by their namespace

Adds the ability to skip article with certain namespace in Wikipedia. These are Wikipedia entries that are images file, audio files, templates, user profiles, ..., etc. The full list can be found here: http://en.wikipedia.org/wiki/Wikipedia:Namespace .

This commit does not change the currently behavior of the WikiCorpus class, unless the additional filter_namespaces argument is passed to the initializer.

For example,

WikiCorpus( inp, lemmatize=lemmatize, filter_namespaces=('0','2') )

will only include main articles and user profile pages into the corpus, while

WikiCorpus( inp, lemmatize=lemmatize )

will be the same as the current behavior, using all entries in the Wikipedia dump.