localwiki/sapling — Issue #23: Refactor HTML parsing/serialization
Author: philipn
Created: 2011-11-24 09:53:50+00:00
State: open

No time to do this work now, so just dumping some thoughts here regarding how to do this.

Here's the code which currently does the sanitization of page contents:

{{{
def sanitize_html_fragment(unsafe, allowed_elements=None):
# TODO: make this more simple / understandable and factor out from
# plugins.html_to_template_text
if not allowed_elements:
allowed_elements = sanitizer.HTMLSanitizer.allowed_elements
p = html5lib.HTMLParser(tree=treebuilders.getTreeBuilder("lxml"),
tokenizer=custom_sanitizer(allowed_elements),
namespaceHTMLElements=False)
top_level_elements = p.parseFragment(unsafe)
# put top level elements in container
container = etree.Element('div')
if top_level_elements and not hasattr(top_level_elements[0], 'tag'):
container.text = top_level_elements.pop(0)
container.extend(top_level_elements)

html_bits = [etree.tostring(elem, method='html', encoding='utf-8')
for elem in container]

return ''.join([escape(container.text or '')] + html_bits)
}}}

As noted, very similar code also exists in pages.plugins where we have to parse the page contents and serialize again.

First, we should factor this out into the field code and make all the things that might be needed accessible (parser, treewalker, serializer, etc).

Then we need to investigate the best framework to use to do this work. Currently we are using lxml.etree, which is full featured and widely used, but there are some things that seem weird about it, such as text nodes being attached to elements as element.text or element.tail. This can lead to bugs and security issues. For example, if you forget to escape the text node before appending it to the output as is done above, the text node (unescaped) will get sent to the browser as regular html code. A mode browser-like DOM view, where text is also a node, would be much better.

In the investigation, I'd take a look at all the tree builder families available in html5lib and pick the best one. Then make sure it can do all of the things we need: parse HTML fragment, sanitization on read, correct HTML output of a fragment, walking through the tree, etc.