nltk/nltk — Issue #178: Double quotes not correctly handled in TreebankTokenizer
Author: alexrudnick
Created: 2012-01-17 06:27:52+00:00
State: open

According to the official Penn Treebank Tokenization page at http://www.cis.upenn.edu/~treebank/tokenization.html, double quotes (") should be changed to doubled single forward- and backward- quotes (`` and ''). However, nltk.word_tokenize() or (TreebankTokenizer) does not follow this rule.

>>> nltk.word_tokenize('He went to the "Concert for Hope" yesterday.')
['He',
'went',
'to',
'the',
'"',
'Concert',
'for',
'Hope',
'"',
'yesterday',
'.']

Migrated from http://code.google.com/p/nltk/issues/detail?id=713