nltk/nltk — Issue #6: Treebank Tokenizer buggy
Author: desilinguist
Created: 2011-12-21 15:35:33+00:00
State: open

The NLTK TreebankWordTokenizer appears to have at least two bugs in it:

(a) It doesn't properly handle sentences with quotes at the end.

Input:
``If we don't have any further suggestions as to how we should present our case, we should be able to finish this.''

Output:
['`', '`', 'If', 'we', 'do', "n't", 'have', 'any', 'further', 'suggestions', 'as', 'to', 'how', 'we', 'should', 'present', 'our', 'case', ',', 'we', 'should', 'be', 'able', 'to', 'finish', "this.''"]

(b) According to the official Penn Treebank Tokenization page at http://www.cis.upenn.edu/~treebank/tokenization.html, double quotes (") should be changed to doubled single forward- and backward- quotes (`` and ''). However, nltk.word_tokenize() (TreebankTokenizer) does not follow this rule.

Input:
He went to the "Concert for Hope" yesterday.

Output:
['He', 'went', 'to', 'the', '"', 'Concert', 'for', 'Hope', '"', 'yesterday', '.']

One of my colleagues, Michael Heilman, took the official Penn Treebank sed script (http://www.cis.upenn.edu/~treebank/tokenizer.sed) and ported it to NLTK using the Tokenizer API. It can be found here: https://gist.github.com/1506443