RDFLib/rdflib — Issue #79: UnicodeEncodeError: 'ascii' codec can't encode character - N3Parser breaks when parsing unicode
Author: gjhiggins
Created: 2012-02-20 08:32:10+00:00
State: open

anthonyg...@gmail.com, 2009-09-24T14:12:35.000Z

What steps will reproduce the problem?

from codecs import getreader
from rdflib.Graph import Graph
from StringIO import StringIO

rdf = getreader('utf-8')(StringIO(u"""@prefix skos:
.
@prefix rdf: .
@prefix : .

:world rdf:type skos:Concept;
skos:prefLabel "World"@en.
:africa rdf:type skos:Concept;
skos:prefLabel "Africa"@en;
skos:broaderTransitive :world.
:CI rdf:type skos:Concept;
skos:prefLabel "C\u00f4te d'Ivoire"@en;
skos:broaderTransitive :africa.
""".encode('utf-8')))

g = Graph()
g.parse(rdf, 'http://www.test.org', 'n3')
print len(g)

What is the expected output? What do you see instead?

Expect - 3
Actual - UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4'
in position 1: ordinal not in range(128)

What version of the product are you using? On what operating system?

rdflib 2.4.2 / Windows Vista 64 bit / Python 2.4.4

Please provide any additional information below.

This breaks in the same way if you pass in a unicode file using
codecs.open('filename', 'r', 'utf-8')
It parses without error if you pass in a unicode file opened using
open('filename', 'r'), however, characters outside ASCII then get double
escaped. e.g. "\\xf4".

Comment 1 by eik...@gmail.com
Added test case in r1714.

Comment 2 by eik...@gmail.com

Comment 3 by eik...@gmail.com

Comment 4 by gromgull

Comment 5 by gromgull
This is fixed for N3 in r1755 - but see issue 108

Comment 6 by eik...@gmail.com
This issue was updated by revision 71ece3659599.

Comment 7 by eik...@gmail.com
This issue was updated by revision ff1300f36d53.

...