nltk/nltk — Issue #93: tokenizer does not work for words containing non-ascii characters (Linux only?)
Author: alexrudnick
Created: 2012-01-17 06:25:55+00:00
State: open

nltk.tokenize does not work with some non-English accented characters -
these characters are treated like punctuation, and become tokens
themselves. E.g.:
>>> nltk.word_tokenize("vier fünf sechs")
['vier', 'f', '\xc3', '\xbc', 'nf', 'sechs']
>>> nltk.word_tokenize(u"vier fünf sechs")
[u'vier', u'f', u'\xc3', u'\xbc', u'nf', u'sechs']
>>>

I'm running Linux, so this might have something to do with the known Linux
Python 2.x issue that character classes (e.g. string.letters; \w) are not
locale sensitive like they should be:
http://bugs.python.org/issue6525

There is a posting on the problem that says that the Punkt tokenizer does work:

http://groups.google.com/group/nltk-users/browse_thread/thread/730d60be0baee773/fa82fa26553619e7?lnk=gst&q=Unicode+Parsing+Problem#fa82fa26553619e7

... and this does work on my system
>>> PunktWordTokenizer().tokenize("vier fünf sechs")
['vier', 'f\xc3\xbcnf', 'sechs']

A quick remedy might be to mention the Punkt tokenizer in the online book.
A more permanent one might be to direct word_tokenizer to be sensitive to
locale (I can't test this because of the Linux/Python 2.x issue just
mentioned), by adding the re.L flag to the code of the treebank tokenizer
(line 58):

http://nltk.googlecode.com/svn/trunk/doc/api/nltk.tokenize.treebank-pysrc.html#TreebankWordTokenizer.tokenize

My python version is:
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2

And NLTK version is:
>>> nltk.__version__
'2.0b5'

Migrated from http://code.google.com/p/nltk/issues/detail?id=477

earlier comments

nedunk said, at 2010-01-08T20:37:22.000Z:

Hi,

I wrote a quick workaround (lines 60 - 61) and added it to
/usr/local/lib/python2.6/dist-packages/nltk-2.0b7-py2.6.egg/nltk/tokenize/treebank.py

57 # Separate most punctuation
58 text = re.sub(r"([^\w\.\'\-\/,&])", r' \1 ', text)
59
60 # Fix missing Unicode support in in re.sub Python <3.0
61 text = re.sub(r"\s([\xaa-\xff])\s", r"\1" ,text)
62
63 # Separate commas if they're followed by space.
64 # (E.g., don't separate 2,500)
65 text = re.sub(r"(,\s)", r' \1', text)

In Python 3 you can set flags which should solve this problem:

text = re.sub(r"([^\w\.\'\-\/,&])", r' \1 ', text, flags=re.UNICODE)
http://docs.python.org/3.1/library/re.html

nedunk said, at 2010-01-10T10:13:45.000Z:

Hi,
trying this:
In [7]: unicode('ö ä ü Ö Ä Ü ß')
Out[7]: u'\xf6 \xe4 \xfc \xd6 \xc4 \xdc \xdf'

I think this is a better range for german special characters (umlaute)

\xc4-\xfc

text = re.sub(r"\s([\xc4-\xfc])\s", r"\1" ,text)

remohammadi said, at 2010-02-04T16:37:11.000Z:

I have the probelm on Mac too. Although the text is Persian.

jan.strunk said, at 2010-11-06T22:24:55.000Z:

As far as I know you can also set the re.UNICODE flag in Python 2.6.

From the Python 2.6 documentation: http://docs.python.org/library/re.html
"re.UNICODE¶

Make \w, \W, \b, \B, \d, \D, \s and \S dependent on the Unicode character properties database.

New in version 2.0."

That's at least what I teach in my Intro to Programming for Computational Linguists...