Repository: Python-Markdown/markdown (python)

Issue:
Hexadecimal HTML entities are not rendered
According to standard Markdown, HTML entities are to be recognized and rendered correctly (https://daringfireball.net/projects/markdown/syntax#autoescape). Up to v2.6.11, Python-Markdown has been able to do so, but with v3 it does not work any longer.

A hex HTML entity is not recognized. The leading ampersand is rendered as "`&amp;`", and in HTML, the entity consequently reads as the entity itself, like, for example `&#x25a0;` instead of the intended ■.

Pull request:
Make ENTITY_RE support hexadecimal entities
Fixes #712.

Patch:
diff --git a/markdown/inlinepatterns.py b/markdown/inlinepatterns.py
index e4af53e..3d3e65f 100644
--- a/markdown/inlinepatterns.py
+++ b/markdown/inlinepatterns.py
@@ -155,8 +155,8 @@ AUTOMAIL_RE = r'<([^> \!]*@[^> ]*)>'
 # <...>
 HTML_RE = r'(\<([a-zA-Z/][^\>]*?|\!--.*?--)\>)'
 
-# &amp;
-ENTITY_RE = r'(&(?:\#[0-9]+|[a-zA-Z0-9]+);)'
+# "&#38;" (decimal) or "&#x26;" (hex) or "&amp;" (named)
+ENTITY_RE = r'(&(?:\#[0-9]+|\#x[0-9a-fA-F]+|[a-zA-Z0-9]+);)'
 
 # two spaces at end of line
 LINE_BREAK_RE = r'  \n'
diff --git a/markdown/serializers.py b/markdown/serializers.py
index 3cfa6bb..20cc45a 100644
--- a/markdown/serializers.py
+++ b/markdown/serializers.py
@@ -53,7 +53,7 @@ __all__ = ['to_html_string', 'to_xhtml_string']
 
 HTML_EMPTY = ("area", "base", "basefont", "br", "col", "frame", "hr",
               "img", "input", "isindex", "link", "meta", "param")
-RE_AMP = re.compile(r'&(?!(?:\#[0-9]+|[0-9a-z]+);)', re.I)
+RE_AMP = re.compile(r'&(?!(?:\#[0-9]+|\#x[0-9a-f]+|[0-9a-z]+);)', re.I)
 
 try:
     HTML_EMPTY = set(HTML_EMPTY)


Test patch:
diff --git a/tests/test_syntax/inline/test_entities.py b/tests/test_syntax/inline/test_entities.py
new file mode 100644
index 0000000..67b0647
--- /dev/null
+++ b/tests/test_syntax/inline/test_entities.py
@@ -0,0 +1,44 @@
+# -*- coding: utf-8 -*-
+"""
+Python Markdown
+
+A Python implementation of John Gruber's Markdown.
+
+Documentation: https://python-markdown.github.io/
+GitHub: https://github.com/Python-Markdown/markdown/
+PyPI: https://pypi.org/project/Markdown/
+
+Started by Manfred Stienstra (http://www.dwerg.net/).
+Maintained for a few years by Yuri Takhteyev (http://www.freewisdom.org).
+Currently maintained by Waylan Limberg (https://github.com/waylan),
+Dmitry Shachnev (https://github.com/mitya57) and Isaac Muse (https://github.com/facelessuser).
+
+Copyright 2007-2018 The Python Markdown Project (v. 1.7 and later)
+Copyright 2004, 2005, 2006 Yuri Takhteyev (v. 0.2-1.6b)
+Copyright 2004 Manfred Stienstra (the original version)
+
+License: BSD (see LICENSE.md for details).
+"""
+
+from markdown.test_tools import TestCase
+
+
+class TestEntities(TestCase):
+
+    def test_named_entities(self):
+        self.assertMarkdownRenders("&amp;", "<p>&amp;</p>")
+        self.assertMarkdownRenders("&sup2;", "<p>&sup2;</p>")
+        self.assertMarkdownRenders("&Aacute;", "<p>&Aacute;</p>")
+
+    def test_decimal_entities(self):
+        self.assertMarkdownRenders("&#38;", "<p>&#38;</p>")
+        self.assertMarkdownRenders("&#178;", "<p>&#178;</p>")
+
+    def test_hexadecimal_entities(self):
+        self.assertMarkdownRenders("&#x00026;", "<p>&#x00026;</p>")
+        self.assertMarkdownRenders("&#xB2;", "<p>&#xB2;</p>")
+
+    def test_false_entities(self):
+        self.assertMarkdownRenders("&not an entity;", "<p>&amp;not an entity;</p>")
+        self.assertMarkdownRenders("&#B2;", "<p>&amp;#B2;</p>")
+        self.assertMarkdownRenders("&#xnothex;", "<p>&amp;#xnothex;</p>")