piskvorky/gensim — pull request #86 by larsmans (2012-07-14 18:51:27+00:00)
Title: Optimize Porter stemming

I took the liberty of optimizing the Porter stemmer, cleaning up the code as I went. The result is roughly 20% faster than the original, as measured by running the script

from gensim.parsing.preprocessing import PorterStemmer

st = PorterStemmer()

for ln in open("/usr/share/dict/american-english-insane"):
st.stem(ln.strip())

where `american-english-insane` is the dictionary file from the Ubuntu package [`wamerican-insane`](http://packages.ubuntu.com/precise/wamerican-insane).

I removed the option the stem a substring of a string; it wasn't being used by Gensim and doesn't really make sense. Callers can take a substring themselves.

The stemmer now also does lowercasing itself.

The comments have been stripped of references to pointers and NUL bytes that only made sense in the context of the original C implementation.

(The code still isn't very Pythonic, but it's closer to that ideal now.)