nltk/nltk — Issue #143: running concordance search for a Portuguese word causes Unicode exception
Author: alexrudnick
Created: 2012-01-17 06:27:03+00:00
State: open

Using NLTK version 2.0b9.

Run this from the command line:

ptext1.concordance('olhos')

Note that this is a line mentioned in one of the online documentation examples ( http://nltk.googlecode.com/svn/trunk/doc/howto/portuguese_en.html ).

Output as follows:

>>> ptext1.concordance('olhos')
Building index...
Displaying 25 of 138 matches:
De pé , à cabeceira da cama , com os olhos estúpidos , a boca entreaberta , a t
orelhas . Pela minha parte fechei os olhos e deixei - me ir à ventura . Já agor
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python26\Lib\site-packages\nltk\text.py", line 314, in concordance
self._concordance_index.print_concordance(word, width, lines)
File "C:\Python26\Lib\site-packages\nltk\text.py", line 189, in print_concorda
nce
print left, word, right
File "C:\Python26\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xf5' in position 1
: character maps to <undefined>

Apparently, the second character in the third line that would be listed in the concordance for the word 'olhos' is causing a problem.

That line (which can be seen in the documentation) looks something like this (except that the accented characters are wrong):

x›es de cŽrebro enfermo . Como ia de olhos fechados , n‹o via o caminho ; lembr

I ran into this when investigating a separate bug (Issue 637) that had to do with the poor display of Portuguese text in the online example.

Migrated from http://code.google.com/p/nltk/issues/detail?id=643

earlier comments

StevenBird1 said, at 2011-03-11T04:57:14.000Z:

I'm unable to replicate this problem (using my mac). The second character in the third line prints fine. What platform are you using?

>>> ptext1.concordance('olhos')
Building index...
Displaying 25 of 138 matches:
De pé , à cabeceira da cama , com os olhos estúpidos , a boca entreaberta , a t
orelhas . Pela minha parte fechei os olhos e deixei - me ir à ventura . Já agor
xões de cérebro enfermo . Como ia de olhos fechados , não via o caminho ; lembr
gelos eternos . Com efeito , abri os olhos e vi que o meu animal galopava numa
...

alanf333 said, at 2011-03-11T17:03:14.000Z:

I see it under both Windows XP and Windows Vista.