RDFLib/rdflib — Issue #154: html entities being stripped when parsing rdfa with DOCTYPE
Author: gjhiggins
Created: 2012-02-20 08:33:59+00:00
State: open

ed.summers, 2011-04-11T16:23:55.000Z

I am not 100% certain if this is the accepted behavior here, but I've noticed that rdflib's RDFa parser is stripping HTML entities when processing XHTML, but that similar processing without the RDFa DOCTYPE results in the entity being translated to UTF-8.

See the attached test: entities.py ...

Comment 1 by ed.summers

Comment 2 by ed.summers

Comment 3 by ed.summers
After some poking around it looks like HTML with a DOCTYPE is successfully parsed by minidom (but the entity is stripped out). But when the HTML lacks a DOCTYPE it fails mindom parsing with a:

xml.parsers.expat.ExpatError: undefined entity: line 4, column 60

and is then parsed by html5lib (where the entity is successfully converted to utf8). So that explains why the entities are stripped out from RDFa HTML with the DOCTYPE and converted to UTF8 when there is no DOCTYPE.

Now to figure out what the fix should be, if any...

Comment 4 by ed.summers
Wondering if maybe we just hand off all html parsing to html5lib and make it an rdflib dependency?

Comment 5 by ed.summers
Looks like xml.minidom has no support for entities:

The following interfaces have no implementation in xml.dom.minidom...Entity
http://docs.python.org/library/xml.dom.minidom.html