Repository: mozilla/bleach

Issue:
linkify is escaping valid HTML tags
Originally noticed here in relation to the `<abbr>` tag: https://github.com/mozilla/bleach/issues/400

linkify is now using a list of valid HTML tags that is largely built up from constants in html5lib, here: https://github.com/mozilla/bleach/blob/master/bleach/html5lib_shim.py#L49

However, these constants don't seem to make up a complete list of valid tags&mdash;in addition to `<abbr>` (which is now being appended to the list by Bleach), there are also multiple other tags missing, including `del`, `ins`, `sub`, `sup`, and `span`.

It would probably be best to replace this combination of html5lib constants with a full list of valid tags. I can do a PR for this.

Discussion:

Patch:
diff --git a/bleach/linkifier.py b/bleach/linkifier.py
index 5d815f8..95baba1 100644
--- a/bleach/linkifier.py
+++ b/bleach/linkifier.py
@@ -85,7 +85,7 @@ class Linker(object):
 
     """
     def __init__(self, callbacks=DEFAULT_CALLBACKS, skip_tags=None, parse_email=False,
-                 url_re=URL_RE, email_re=EMAIL_RE):
+                 url_re=URL_RE, email_re=EMAIL_RE, recognized_tags=html5lib_shim.HTML_TAGS):
         """Creates a Linker instance
 
         :arg list callbacks: list of callbacks to run when adjusting tag attributes;
@@ -101,6 +101,9 @@ class Linker(object):
 
         :arg re email_re: email matching regex
 
+        :arg list-of-strings recognized_tags: the list of tags that linkify knows about;
+            everything else gets escaped
+
         :returns: linkified text as unicode
 
         """
@@ -113,7 +116,7 @@ class Linker(object):
         # Create a parser/tokenizer that allows all HTML tags and escapes
         # anything not in that list.
         self.parser = html5lib_shim.BleachHTMLParser(
-            tags=html5lib_shim.HTML_TAGS,
+            tags=recognized_tags,
             strip=False,
             consume_entities=True,
             namespaceHTMLElements=False,
diff --git a/docs/linkify.rst b/docs/linkify.rst
index d60e17b..b8e7884 100644
--- a/docs/linkify.rst
+++ b/docs/linkify.rst
@@ -12,13 +12,14 @@ For example, you could pass in text and have all URL things converted into
 HTML links.
 
 It works by parsing the text as HTML and building a document tree. In this
-way, it's guaranteed never to do weird things to URLs in attribute values,
-can modify the value of attributes on ``<a>`` tags and can even do things
-like skip ``<pre>`` sections.
+way, you're guaranteed to get valid HTML back without weird things like
+having URLs in tag attributes getting linkified.
 
-If you plan to sanitize/clean the text and linkify it, you should do that
-in a single pass using :ref:`LinkifyFilter <linkify-LinkifyFilter>`. This
-is faster and it'll use the list of allowed tags from clean.
+.. note::
+
+   If you plan to sanitize/clean the text and linkify it, you should do that
+   in a single pass using :ref:`LinkifyFilter <linkify-LinkifyFilter>`. This
+   is faster and it'll use the list of allowed tags from clean.
 
 .. note::
 
@@ -297,8 +298,8 @@ writing callbacks that may need to behave differently if the protocol is
 Using ``bleach.linkifier.Linker``
 =================================
 
-If you're linking a lot of text and passing the same argument values or you want
-more configurability, consider using a :py:class:`bleach.linkifier.Linker`
+If you're linking a lot of text and passing the same argument values or you
+need more configurability, consider using a :py:class:`bleach.linkifier.Linker`
 instance.
 
 .. doctest::
@@ -325,8 +326,8 @@ Using ``bleach.linkifier.LinkifyFilter``
 the ``bleach.linkifier.LinkifyFilter`` when walking the tree and serializing it
 back into text.
 
-You can use this filter wherever you can use an html5lib Filter. For example, you
-could use it with ``bleach.Cleaner`` to clean and linkify in one step.
+You can use this filter wherever you can use an html5lib Filter. This lets you
+use it with ``bleach.Cleaner`` to clean and linkify in one step.
 
 For example, using all the defaults:
 


Test patch:
diff --git a/tests/test_linkify.py b/tests/test_linkify.py
index eeea3e3..d29a5c8 100644
--- a/tests/test_linkify.py
+++ b/tests/test_linkify.py
@@ -625,6 +625,23 @@ def test_email_re_arg():
     )
 
 
+def test_recognized_tags_arg():
+    """Verifies that recognized_tags works"""
+    # The html parser doesn't recognize "sarcasm" as a tag, so it escapes it
+    linker = Linker(recognized_tags=['p'])
+    assert (
+        linker.linkify('<p>http://example.com/</p><sarcasm>') ==
+        '<p><a href="http://example.com/" rel="nofollow">http://example.com/</a></p>&lt;sarcasm&gt;'  # noqa
+    )
+
+    # The html parser recognizes "sarcasm" as a tag and fixes it
+    linker = Linker(recognized_tags=['p', 'sarcasm'])
+    assert (
+        linker.linkify('<p>http://example.com/</p><sarcasm>') ==
+        '<p><a href="http://example.com/" rel="nofollow">http://example.com/</a></p><sarcasm></sarcasm>'  # noqa
+    )
+
+
 def test_linkify_idempotent():
     dirty = '<span>invalid & </span> < extra http://link.com<em>'
     assert linkify(linkify(dirty)) == linkify(dirty)