nltk/nltk — Issue #139: Porter stemmer's output inconsistent with that of reference implementations
Author: alexrudnick
Created: 2012-01-17 06:26:57+00:00
State: open

I recently used the NLTK's Porter stemmer and discovered some disrepancies between its output and that of another version of the Porter stemmer that I have used. Following up on these discrepancies, I discovered that there may be some problems with the NLTK's implementation.

There are various reference implementations of the Porter stemmer collected by Martin Porter himself here:

http://tartarus.org/~martin/PorterStemmer/

I tried the one for Ruby to sanity check the NLTK (it takes a word from standard out and spits out its stemmed form immediately, also on standard out):

scripts $ ruby porter_stemmer.rb
shiny
shini

I checked this against the NLTK:

scripts $ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import nltk
nltk.stem.porter.PorterStemmer().stem_word('shiny')

'shini'

So far, so good. And if I compare these results to the expected results (output.txt) provided on the same page for a sample user dict file (voc.txt), they look right:

scripts $ egrep -n '^shiny$' voc.txt
18333:shiny
scripts $ egrep -n '^shini$' output.txt
18333:shini

But if I systematically compare the NLTK results against those in the expected results file, I see a lot of discrepancies (format: word, proper expected stemming, bad NLTK stemming marked by asterisk):

scripts $ ./show_bad_stemming_in_nltk.py voc.txt output.txt
abbey abbei *abbey
abbeys abbei *abbey
abed ab *abe
absey absei *absey
[...]
wrying wry *wri
yesterday yesterdai *yesterday
yesterdays yesterdai *yesterday
yongrey yongrei *yongrey

Spot checking against the Ruby reference implemenation confirms that the NLTK results are in fact the problem:

scripts $ ruby porter_stemmer.rb
abbey
abbei
abbeys
abbei
abed
ab
absey
absei
wrying
wry
yesterday
yesterdai
yesterdays
yesterdai
yongrey
yongrei

I've attached the full list of words for which the NLTK's Porter stemmer provides unexpected results.

(This bug pertains to version 2.0b9. I suspect it exists in all previous versions, but I haven't confirmed that.)

Migrated from http://code.google.com/p/nltk/issues/detail?id=625

earlier comments

gregg.lind said, at 2011-02-09T20:22:57.000Z:

I would be happy to take this one on (by friday), if no one else wants it. I have used this module a lot.

Gregg Lind

StevenBird1 said, at 2011-02-14T07:14:49.000Z:

Attached is Stuart Robinson's fixes, sent to nltk-dev. This seems to be further edits to the previous version rather than a fresh port of the Ruby version as originally discussed. Before this new version can be incorporated we need a set of test cases added to test/stem.doctest.