gjhiggins/rdflib — Issue #83: UnicodeEncodeError: 'ascii' codec can't encode character - N3Parser breaks when parsing unicode
Author: gjhiggins
Created: 2011-10-19 01:15:23+00:00
State: open

Originally filed by anthonyg...@gmail.com on 2009-09-24T14:12:35

What steps will reproduce the problem?

from codecs import getreader
from rdflib.Graph import Graph
from StringIO import StringIO

rdf = getreader('utf-8')(StringIO(u"""@prefix skos:
.
@prefix rdf: .
@prefix : .

:world rdf:type skos:Concept;
skos:prefLabel "World"@en.
:africa rdf:type skos:Concept;
skos:prefLabel "Africa"@en;
skos:broaderTransitive :world.
:CI rdf:type skos:Concept;
skos:prefLabel "C\u00f4te d'Ivoire"@en;
skos:broaderTransitive :africa.
""".encode('utf-8')))

g = Graph()
g.parse(rdf, 'http://www.test.org', 'n3')
print len(g)

What is the expected output? What do you see instead?

Expect - 3
Actual - UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4'
in position 1: ordinal not in range(128)

What version of the product are you using? On what operating system?

rdflib 2.4.2 / Windows Vista 64 bit / Python 2.4.4

Please provide any additional information below.

This breaks in the same way if you pass in a unicode file using
codecs.open('filename', 'r', 'utf-8')
It parses without error if you pass in a unicode file opened using
open('filename', 'r'), however, characters outside ASCII then get double
escaped. e.g. "\\xf4".