nltk/nltk — Issue #65: Text throws UnicodeEncodeError
Author: alexrudnick
Created: 2012-01-17 06:25:14+00:00
State: open

I understand that the Text class is intended mainly for demo and learning
purposes, but I just spent some time debugging my program which happened to
use it before realizing that as architected it doesn't support unicode
text. I can work around, but wanted to let you know. Thanks for a great
toolkit!

What version of NLTK are you using? (See nltk.__version__). Please only
submit bug reports for the current version.
0.9.8

What steps will reproduce the problem? (e.g. include Python source code)
import nltk
unicode_text = u'\u015bwiatowej'
tokens = nltk.word_tokenize(unicode_text)
text = nltk.Text(tokens)

Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/lib/python2.5/site-packages/nltk/text.py", line 283, in __init__
self.name = " ".join(map(str, tokens[:8])) + "..."
UnicodeEncodeError: 'ascii' codec can't encode character u'\u015b' in
position 0
: ordinal not in range(128)

Migrated from http://code.google.com/p/nltk/issues/detail?id=370

earlier comments

arthit said, at 2009-05-02T17:36:15.000Z:

this code will work (2nd line added):

unicode_text = u'\u015bwiatowej'
unicode_text = unicode_text.encode("utf-8")
tokens = nltk.word_tokenize(unicode_text)
text = nltk.Text(tokens)

StevenBird1 said, at 2009-05-03T09:19:48.000Z:

Thanks for pointing this out; its an instance of a more general issue with NLTK,
namely ensuring that everything works right with Unicode strings. We probably won't
do this until NLTK 3.0 (to work with Python 3.0).

robfelty said, at 2009-11-02T17:31:57.000Z:

The above code by Arthit does not work as I would expect it using 2.0b6. This is what
I get (with python 2.6.2 on Ubuntu):
>>> unicode_text = u'\u015bwiatowej'
>>> unicode_text = unicode_text.encode("utf-8")
>>> tokens = nltk.word_tokenize(unicode_text)
>>> tokens
['\xc5', '\x9b', 'wiatowej']

Getting rid of the encode part, I get:
>>> text = u'\u015bwiatowej'
>>> tokens = nltk.word_tokenize(text)
>>> tokens
[u'\u015b', u'wiatowej']

pingelsan said, at 2010-09-25T15:20:18.000Z:

I had a similar problem (with German text).
For your case, this worked for me:

>>> unicode_text = u'\u015bwiatowej'
>>> unicode_text
u'\u015bwiatowej'
>>> unicode_text = unicode_text.encode('utf-8')
>>> unicode_text
'\xc5\x9bwiatowej'
>>> from nltk import PunktWordTokenizer
>>> tokens = PunktWordTokenizer().tokenize(unicode_text)
>>> tokens
['\xc5\x9bwiatowej']
>>> text = nltk.Text(tokens)
>>> text

>>>

Is it safe to assume that most thing work in UTF-8, but in some cases - like the tokenizer here - we have to use some kind of drop-in-replacement?