nltk/nltk — Issue #8: TreebankWordTokenizer with unicode characters
Author: aboSamoor
Created: 2011-12-27 20:20:34+00:00
State: open

It seems there is a problem in the TreebankWordTokenizer when it comes to unicode characters as it split any non English word depending on the non ASCII characters! here is an example
**Statement**
In Düsseldorf I took my hat off. But I can't put it back on.

**NLTK**
In D ü sseldorf I took my hat off .
But I ca n't put it back on .

**Stanford (PennTreebank) PTBLexer**
[In, Düsseldorf, I, took, my, hat, off, .]
[But, I, ca, n't, put, it, back, on, .]

**http://www.cis.upenn.edu/~treebank/tokenizer.sed**
In Düsseldorf I took my hat off .
But I ca n't put it back on .