Repository: django-cms/django-cms

Issue:
Output valid sanitized HTML instead of XHTML
Fixes conversion of &lt;div class="something"&gt;&lt;/div&gt; -&gt; &lt;div class="something" /&gt; which isn't valid HTML5.

I need to write some unit tests for this still but wanted to run this by the devs first.  



Discussion:
definitely needs tests

This affects other tags (specially iframes from youtube/vimeo video embedding)

@yakky iframes should be impossible for a while. they're not supported by the text plugin.

@ojii Yes not in standard text plugin, but I extended it to support iframe. Anyway HTMLSanitizer behaviour affects any empty tag

I'm writing a test for this, hope to attach it soon

Trivial test written
Look like the proposed code doesn't work, though; it triggers some weird behaviour in html5lib sanitizer (reproduced with sample html5lib code below, so definitely not a django-cms issue): toke "data" attribute is expected to be a dict by serializer and a list by sanitizer.
@gmcguire which version of html5lib do you use with the above code?

```
import html5lib
from html5lib import treebuilders, treewalkers, serializer
from html5lib.filters import sanitizer

p = html5lib.HTMLParser(tree=treebuilders.getTreeBuilder("dom"))

dom_tree = p.parse("<p class='hi'><strong>Hello</strong> World</p>")

walker = treewalkers.getTreeWalker("dom")

stream = walker(dom_tree)

s = serializer.htmlserializer.HTMLSerializer(sanitize=True,omit_optional_tags=False, quote_attr_values=True)
output_generator = s.serialize(stream)

for item in output_generator:
    print item
```

@yakky `pip freeze | grep html5lib` reports `html5lib==0.90`.  

Thanks for taking on what I haven't had time to do.

I'm not sure I understand your question about `data`.  The `data` variable should be any filelike object or string, according to html5lib's docs: http://code.google.com/p/html5lib/source/browse/python/html5lib/html5parser.py#236  I believe Django CMS passes a string to the `clean_html` method.  It's only used once, to build the DOM tree object.  After that the code builds a tree walker for that DOM, and then walks the tree, re-serializing the DOM and sanitizing as it goes.

Your code above produces this output for me:

``` html
<html>
<head>
</head>
<body>
<p class="hi">
<strong>
Hello
</strong>

World
</p>
</body>
</html>
```

@gmcguire so it looks like html5lib is definitely the culprit.
Current version html5lib is 0.95 and it's affected by what seems like a bug.
`data` issue is strictly a html5lib problem: at least 0.95 version expects different `data` type in sanitizer and serializer, but ``data` is generated by the html5lib parser; it's not a django-cms issue.
I'll try to contact html5lib mantainers to investigate this

Updated: I remembred the html5lib version wrong: it's 0.95 not 0.97. Corrected above

I tried a slightly different approach that works with 0.95 version of html5lib: nephila/django-cms@29ed8fda849605da080e4be4903cde341133bee9
Works on 0.90 version too

Could this be considered for 2.3.1?

does this actually affect the website in the real world? Or does it just output 'invalid' HTML?

Outputs valid XHTML, but invalid HTML{4,5}.
And yes, this affects live websites: we came up with an hackish solution, but this is definitely cleaner and proper

From what I can tell the Text plugin runs through this code when you save an entry, I think to prevent script tags and other malicious entries.  If you're using XHTML for your base template this output is just fine.  If you're using HTML5 though, the output is not valid.

"And yes, this affects live websites: we came up with an hackish solution, but this is definitely cleaner and proper"

In what way? AFAIK browsers are insanely good at working with invalid HTML.

@gmcguire correct, it filters malicious parts in your HTML out. This is why there's no way to disable this.

Recent Chrome/Safari versions (I don't know about FF or IE) seem to treat `<div/>` as an opening DIV, and continue parsing until they find `</div>`.  This causes all kinds of layout issues as you might imagine. 

In our site, we have lots of `<div class="section-break"></div>` to style breaks between content sections.  These become `<div class="section-break"/>` when you edit/save a Text plugin instance that contains one, and causes everything that follows to become a child of the div instead of a sibling.

@ojii Are good in all but this case.
As long as you declare HTML5 doctype (haven't tried with HTML4) they treat `<div />` as a `<div>` and keep on looking form a closing `</div>`
This happens for any non-void element I tried


Resolving patch:
diff --git a/cms/utils/html.py b/cms/utils/html.py
--- a/cms/utils/html.py
+++ b/cms/utils/html.py
@@ -1,27 +1,9 @@
 # -*- coding: utf-8 -*-
+from html5lib import sanitizer, serializer, treebuilders, treewalkers
 import html5lib
-from html5lib import sanitizer
 
-
-DEFAULT_PARSER = html5lib.HTMLParser(tokenizer=sanitizer.HTMLSanitizer)
-
-
-def _get_inner_body(doc):
-    # find 'body'
-    def _rec(node):
-        if node.type == 5: # Element Type
-            if node.name == 'body': # the body element
-                return node
-        for child in node.childNodes:
-            childfound = _rec(child)
-            if childfound:
-                return childfound
-        return None
-    body = _rec(doc)
-    out = []
-    for node in body.childNodes:
-        out.append(node.toxml())
-    return u''.join(out)
+DEFAULT_PARSER = html5lib.HTMLParser(tokenizer=sanitizer.HTMLSanitizer,
+                                     tree=treebuilders.getTreeBuilder("dom"))
 
 def clean_html(data, full=True, parser=DEFAULT_PARSER):
     """
@@ -30,8 +12,12 @@ def clean_html(data, full=True, parser=DEFAULT_PARSER):
     If full is False, only the contents inside <body> will be returned (without
     the <body> tags).
     """
-    doc = parser.parse(data)
     if full:
-        return doc.toxml()
+        dom_tree = parser.parse(data)
     else:
-        return _get_inner_body(doc)
+        dom_tree = parser.parseFragment(data)
+    walker = treewalkers.getTreeWalker("dom")
+    stream = walker(dom_tree)
+    s = serializer.htmlserializer.HTMLSerializer(omit_optional_tags=False,
+                                                 quote_attr_values=True)
+    return u''.join(s.serialize(stream))


Test patch:
diff --git a/cms/tests/plugins.py b/cms/tests/plugins.py
--- a/cms/tests/plugins.py
+++ b/cms/tests/plugins.py
@@ -142,6 +142,62 @@ def test_add_cancel_plugin(self):
         self.assertEquals(response.status_code, 200)
         self.assertEquals(0, Text.objects.count())
 
+    def test_add_text_plugin_empty_tag(self):
+        """
+        Test that you can add a text plugin
+        """
+        # add a new text plugin
+        page_data = self.get_new_page_data()
+        response = self.client.post(URL_CMS_PAGE_ADD, page_data)
+        page = Page.objects.all()[0]
+        plugin_data = {
+            'plugin_type':"TextPlugin",
+            'language':settings.LANGUAGES[0][0],
+            'placeholder':page.placeholders.get(slot="body").pk,
+            }
+        response = self.client.post(URL_CMS_PLUGIN_ADD, plugin_data)
+        self.assertEquals(response.status_code, 200)
+        self.assertEquals(int(response.content), CMSPlugin.objects.all()[0].pk)
+        # now edit the plugin
+        edit_url = URL_CMS_PLUGIN_EDIT + response.content + "/"
+        response = self.client.get(edit_url)
+        self.assertEquals(response.status_code, 200)
+        data = {
+            "body":'<div class="someclass"></div><p>foo</p>'
+        }
+        response = self.client.post(edit_url, data)
+        self.assertEquals(response.status_code, 200)
+        txt = Text.objects.all()[0]
+        self.assertEquals('<div class="someclass"></div><p>foo</p>', txt.body)
+
+    def test_add_text_plugin_html_sanitizer(self):
+        """
+        Test that you can add a text plugin
+        """
+        # add a new text plugin
+        page_data = self.get_new_page_data()
+        response = self.client.post(URL_CMS_PAGE_ADD, page_data)
+        page = Page.objects.all()[0]
+        plugin_data = {
+            'plugin_type':"TextPlugin",
+            'language':settings.LANGUAGES[0][0],
+            'placeholder':page.placeholders.get(slot="body").pk,
+            }
+        response = self.client.post(URL_CMS_PLUGIN_ADD, plugin_data)
+        self.assertEquals(response.status_code, 200)
+        self.assertEquals(int(response.content), CMSPlugin.objects.all()[0].pk)
+        # now edit the plugin
+        edit_url = URL_CMS_PLUGIN_EDIT + response.content + "/"
+        response = self.client.get(edit_url)
+        self.assertEquals(response.status_code, 200)
+        data = {
+            "body":'<script>var bar="hacked"</script>'
+        }
+        response = self.client.post(edit_url, data)
+        self.assertEquals(response.status_code, 200)
+        txt = Text.objects.all()[0]
+        self.assertEquals('&lt;script&gt;var bar="hacked"&lt;/script&gt;', txt.body)
+
     def test_copy_plugins(self):
         """
         Test that copying plugins works as expected.