nltk/nltk — pull request #303 by ExplodingCabbage (2012-10-28 23:47:12+00:00)
Title: A slew of improvements to the Porter Stemmer

Hi,

I'm new to GitHub so sorry in advance if I'm doing anything wrong or improper here. Drop me a message if so and I'll fix it.

I've made some changes to the Porter Stemmer. The original version of it was ported from C and so there was lots of weirdness going on. Basically, as detailed in the commit message, my changes were:

1) Making it threadsafe
2) Making it faster (~ 40%)
3) Making it cleaner / more Pythonic

I wrote up a consistency test and a performance test to confirm that this version spits out the same results and is faster, and ran this against the complete text of the King James Bible (804875 words). It passed the consistency test (i.e. each word stemmed to the same stem using either my improved stemmer or the old one), and on my laptop the old stemmer takes 15.9 seconds while the new stemmer takes 9.6 seconds.

Would be very cool if you'd pull my changes. :)

Regards,

Cabbage