nltk/nltk — Issue #141: bad display of Portuguese example on website page
Author: alexrudnick
Created: 2012-01-17 06:27:00+00:00
State: open

Using current version of NLTK: 2.0b9

On this website page, the Portuguese text is messed up:
http://nltk.googlecode.com/svn/trunk/doc/howto/portuguese_en.html

I took a look at the HTML source. There are two problems with it:
(1) the encoding is ASCII(?!): <?xml version="1.0" encoding="ascii" ?>
(2) the entity numbers are wrong.

For example, this line:

<span class="pysrc-output">ptext1: Mem&#151;rias P&#151;stumas de Br&#135;s Cubas (1881)</span>

should read:

<span class="pysrc-output">ptext1: Mem&#243;rias P&#243;stumas de Br&#225;s Cubas (1881)</span>

I could fix this for you. For instance, I could attach a file where these problems were cleaned up. Just let me know.

Later on the page, there are cases like these:
['o', 'amor', 'da', 'gl\xf3ria', 'era', 'a', 'coisa', 'mais', ...

There are undoubtedly more user-friendly ways to display these strings, but I'm still investigating.

Migrated from http://code.google.com/p/nltk/issues/detail?id=637

earlier comments

StevenBird1 said, at 2011-03-07T00:10:45.000Z:

Thank you. Corrections to the source file would be welcome:
http://nltk.googlecode.com/svn/trunk/nltk/nltk/test/portuguese_en.doctest

alanf333 said, at 2011-03-11T01:13:25.000Z:

Replacement of part of the text should be straightforward. However, as I was investigating this, I ran into an error. Executing this line:

ptext1.concordance('olhos')

caused this output:

>>> ptext1.concordance('olhos')
Building index...
Displaying 25 of 138 matches:
De pé , à cabeceira da cama , com os olhos estúpidos , a boca entreaberta , a t
orelhas . Pela minha parte fechei os olhos e deixei - me ir à ventura . Já agor
Traceback (most recent call last):
File "", line 1, in
File "C:\Python26\Lib\site-packages\nltk\text.py", line 314, in concordance
self._concordance_index.print_concordance(word, width, lines)
File "C:\Python26\Lib\site-packages\nltk\text.py", line 189, in print_concorda
nce
print left, word, right
File "C:\Python26\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xf5' in position 1
: character maps to

Apparently, the second character in the third line that would be listed in the concordance for the word 'olhos' is causing a problem. I'll report this as a separate bug.

alanf333 said, at 2011-03-11T17:24:43.000Z:

Back to the core issue: I'm not sure that simply changing the doctest file will be enough to fix the display on the website unless the XML processing instruction that specifies an ASCII encoding is changed. How is the doctest file being processed to produce the (X)HTML for the website?

alanf333 said, at 2011-07-17T16:20:03.000Z:

I let this thread drop in March because I was busy with classwork, but now I have some time to revisit it. How would I convert the doctest file to html in order to verify the output? Is there some way I can control the XML processing instruction that specifies an ASCII encoding?

alanf333 said, at 2011-07-17T16:27:02.000Z:

Note that when one runs the output, the actual runtime display for ptext1 and ptext2 (Windows Vista/Python 2.6/NLTK 2.0b9), as opposed to the way the display is depicted in the HTML, is fine:

>>> from nltk.examples.pt import *
*** Introductory Examples for the NLTK Book ***
Loading ptext1, ... and psent1, ...
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
ptext1: Memórias Póstumas de Brás Cubas (1881)
ptext2: Dom Casmurro (1899)
ptext3: Gênesis
ptext4: Folha de Sau Paulo (1994)
>>> ptext2