gjhiggins/rdflib — Issue #129: NT output encoding to utf-8 broken as it goes through _xmlcharrefreplace
Author: gjhiggins
Created: 2011-10-19 01:28:28+00:00
State: open

Originally filed by joernh...@gmail.com on 2010-07-09T15:10:49

Hi,

if a graph is serialized in NT format and encoding="utf-8", I would expect the output to be utf-8. Instead the output includes a lot "asciified" unicode chars for each non-ascii-char, i.e., all occurrences of Umlauts are replaced with strange "\uXXXX" strings, as it would work in XML output, where this is an escape string for unicode chars.

The reason for this possibly is that the _xmlcharrefreplace function is called in http://code.google.com/p/rdflib/source/browse/trunk/rdflib/plugins/serializers/nt.py?r=1883 (line 29), which substitutes all unicode chars with a ascii string starting with "\u" and then 4 ascii letters for the unicode's hex code.

I think it is strange to do such xml-things in an NT file. Also I spottet a problem in the _xmlcharrefreplace function: it limits the output to just 4 ascii letters representing hex for a 16 bit unicode, while unicode can have more, e.g., for chars in the astral planes: http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF

Instead of using '_xmlcharref_encode(triple[2].n3())' in line 29, a simple 'triple[2].n3()' does the job for me, but perhaps someone who's more into the interna of rdflib could have a look.

Jörn