CVE-2018-19787: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting') (CWE-79)
value: An issue was discovered in lxml before 4.2.5. lxml/html/clean.py in the lxml.html.clean module does not remove javascript: URLs that use escaping, allowing a remote attacker to conduct XSS attacks, as demonstrated by "j a v a s c r i p t:" in Internet Explorer. This is a similar issue to CVE-2014-3146.
CWE description: The product does not neutralize or incorrectly neutralizes user-controllable input before it is placed in output that is used as a web page that is served to other users.
CVSS v3 base score: 6.1
Published: 2018-12-02T10:29Z
Repository: https://github.com/lxml/lxml
Language: Python
Files changed: src/lxml/html/clean.py, src/lxml/html/tests/test_clean.txt
Commit date: 2018-09-09 16:44:17 +0200
Commit message: Fix: make the cleaner also remove javascript URLs that use escaping.

Diff: diff --git a/src/lxml/html/clean.py b/src/lxml/html/clean.py
index adc3f450..11da2958 100644
--- a/src/lxml/html/clean.py
+++ b/src/lxml/html/clean.py
@@ -6,13 +6,14 @@ details.
 
 import re
 import copy
 try:
     from urlparse import urlsplit
+    from urllib import unquote_plus
 except ImportError:
     # Python 3
-    from urllib.parse import urlsplit
+    from urllib.parse import urlsplit, unquote_plus
 from lxml import etree
 from lxml.html import defs
 from lxml.html import fromstring, XHTML_NAMESPACE
 from lxml.html import xhtml_to_html, _transform_result
 
@@ -480,11 +481,11 @@ class Cleaner(object):
         for el in bad:
             el.drop_tree()
 
     def _remove_javascript_link(self, link):
         # links like "j a v a s c r i p t:" might be interpreted in IE
-        new = _substitute_whitespace('', link)
+        new = _substitute_whitespace('', unquote_plus(link))
         if _is_javascript_scheme(new):
             # FIXME: should this be None to delete?
             return ''
         return link
 
diff --git a/src/lxml/html/tests/test_clean.txt b/src/lxml/html/tests/test_clean.txt
index c78ab4f1..2824f64c 100644
--- a/src/lxml/html/tests/test_clean.txt
+++ b/src/lxml/html/tests/test_clean.txt
@@ -16,11 +16,11 @@
 ...     </style>
 ...   </head>
 ...   <body onload="evil_function()">
 ...     <!-- I am interpreted for EVIL! -->
 ...     <a href="javascript:evil_function()">a link</a>
-...     <a href="j\x01a\x02v\x03a\x04s\x05c\x06r\x07i\x0Ep t:evil_function()">a control char link</a>
+...     <a href="j\x01a\x02v\x03a\x04s\x05c\x06r\x07i\x0Ep t%20:evil_function()">a control char link</a>
 ...     <a href="data:text/html;base64,PHNjcmlwdD5hbGVydCgidGVzdCIpOzwvc2NyaXB0Pg==">data</a>
 ...     <a href="#" onclick="evil_function()">another link</a>
 ...     <p onclick="evil_function()">a paragraph</p>
 ...     <div style="display: none">secret EVIL!</div>
 ...     <object> of EVIL! </object>
@@ -49,11 +49,11 @@
     </style>
   </head>
   <body onload="evil_function()">
     <!-- I am interpreted for EVIL! -->
     <a href="javascript:evil_function()">a link</a>
-    <a href="javascrip t:evil_function()">a control char link</a>
+    <a href="javascrip t%20:evil_function()">a control char link</a>
     <a href="data:text/html;base64,PHNjcmlwdD5hbGVydCgidGVzdCIpOzwvc2NyaXB0Pg==">data</a>
     <a href="#" onclick="evil_function()">another link</a>
     <p onclick="evil_function()">a paragraph</p>
     <div style="display: none">secret EVIL!</div>
     <object> of EVIL! </object>
@@ -82,11 +82,11 @@
     </style>
   </head>
   <body onload="evil_function()">
     <!-- I am interpreted for EVIL! -->
     <a href="javascript:evil_function()">a link</a>
-    <a href="javascrip%20t:evil_function()">a control char link</a>
+    <a href="javascrip%20t%20:evil_function()">a control char link</a>
     <a href="data:text/html;base64,PHNjcmlwdD5hbGVydCgidGVzdCIpOzwvc2NyaXB0Pg==">data</a>
     <a href="#" onclick="evil_function()">another link</a>
     <p onclick="evil_function()">a paragraph</p>
     <div style="display: none">secret EVIL!</div>
     <object> of EVIL! </object>

Vulnerable code:     from urllib.parse import urlsplit
        new = _substitute_whitespace('', link)
...     <a href="j\x01a\x02v\x03a\x04s\x05c\x06r\x07i\x0Ep t:evil_function()">a control char link</a>
    <a href="javascrip t:evil_function()">a control char link</a>
    <a href="javascrip%20t:evil_function()">a control char link</a>
Fixed code:     from urllib import unquote_plus
    from urllib.parse import urlsplit, unquote_plus
        new = _substitute_whitespace('', unquote_plus(link))
...     <a href="j\x01a\x02v\x03a\x04s\x05c\x06r\x07i\x0Ep t%20:evil_function()">a control char link</a>
    <a href="javascrip t%20:evil_function()">a control char link</a>
    <a href="javascrip%20t%20:evil_function()">a control char link</a>