piskvorky/gensim — pull request #87 by larsmans (2012-07-19 21:18:11+00:00)
Title: Proper XML parser for Wikipedia dumps

I've reworked the Wikipedia dump parser in `gensim.corpora.wikicorpus` to use a proper XML parser, instead of the line-oriented parser used now. Experiments on smaller versions of Wikipedia, such as the Norwegian one, show that parsing speed is roughly unchanged.

The main benefit of this approach is that articles can be pruned earlier, e.g. based on the title or the namespace (a user may want to consider only articles from the main namespace, with titles not matching `^List of`, etc.). I think there is also some more room for optimization.

Please let me know what you think.