Repository: Python-Markdown/markdown (python)

Issue:
Heading names in toc_tokens contain stashed HTML placeholders
If a Markdown heading contains HTML, the corresponding entry in the `.toc_tokens` property ends up with HTML placeholders when returned to the user. The following example should illustrate the problem:

```
>>> import markdown
>>> md = markdown.Markdown(extensions=['toc'])
>>> md.convert('# <code>Heading</code>\n')
'<h1 id="heading"><code>Heading</code></h1>'
>>> md.toc_tokens
[{'level': 1, 'id': 'heading', 'name': '\x02wzxhzdk:0\x03Heading\x02wzxhzdk:1\x03', 'children': []}]
```

While this isn't *too* hard to fix (we could just un-stash the HTML immediately before returning it to the user), it does raise a bigger question: what should the data format of the `name` field in `toc_tokens` be? Is it...

* Markdown (so the value would be exactly as in the source file: `<code>Heading</code>`)
* Plain text (so the value would strip HTML: `Heading`)
* HTML (similar to the Markdown format, but with HTML entities replaced, so `<code>a>b</code>` becomes `<code>a&gt;b</code>`)

In particular, this is relevant for https://github.com/mkdocs/mkdocs/pull/1970. Prior to that PR, MkDocs would build an internal representation of the TOC by parsing the HTML from `.toc`. With the change, it (tries to) use `.toc_tokens`, but fails due to this issue.

FWIW, I think MkDocs wants this to be plain text in the end, but HTML makes the most sense to me in general: after all, the purpose of this lib is to convert Markdown to HTML. (MkDocs would then just need to parse the HTML fragment and strip out the tags.)

Pull request:
Don't emit stashed HTML tag placeholders in `.toc_tokens`
This should resolve #899. The fix is a bit subtle, since we want to emit HTML entities for `.toc_tokens`, but not HTML tags. We also want to be careful to clean up `data-toc-label` as needed. Note: this change means that raw HTML in a heading is no longer passed through to the HTML for `.toc`.

I'm not *entirely* happy with this. In practice, I think the ideal would be for the toc extension to include the (HTML-ized) Markdown from a heading in its TOC entry, and to have `.toc_tokens` include both `html` and `plaintext` fields. This patch gives us a sort of middle ground, where we just use the HTML with tags stripped. The "ideal" is a larger behavioral change though, and I'm not sure it makes sense to do that for a minor release...

If you think it makes sense to implement the "ideal" solution now, I can try and do that. However, this patch is sufficient for what MkDocs needs.

Patch:
diff --git a/markdown/extensions/toc.py b/markdown/extensions/toc.py
index d490a06..e42836a 100644
--- a/markdown/extensions/toc.py
+++ b/markdown/extensions/toc.py
@@ -43,7 +43,7 @@ def unique(id, ids):
     return id
 
 
-def stashedHTML2text(text, md):
+def stashedHTML2text(text, md, strip_entities=True):
     """ Extract raw HTML from stash, reduce to plain text and swap with placeholder. """
     def _html_sub(m):
         """ Substitute raw html with plain text. """
@@ -51,8 +51,11 @@ def stashedHTML2text(text, md):
             raw = md.htmlStash.rawHtmlBlocks[int(m.group(1))]
         except (IndexError, TypeError):  # pragma: no cover
             return m.group(0)
-        # Strip out tags and entities - leaveing text
-        return re.sub(r'(<[^>]+>)|(&[\#a-zA-Z0-9]+;)', '', raw)
+        # Strip out tags and/or entities - leaving text
+        res = re.sub(r'(<[^>]+>)', '', raw)
+        if strip_entities:
+            res = re.sub(r'(&[\#a-zA-Z0-9]+;)', '', res)
+        return res
 
     return HTML_PLACEHOLDER_RE.sub(_html_sub, text)
 
@@ -259,7 +262,9 @@ class TocTreeprocessor(Treeprocessor):
                 toc_tokens.append({
                     'level': int(el.tag[-1]),
                     'id': el.attrib["id"],
-                    'name': el.attrib.get('data-toc-label', text)
+                    'name': unescape(stashedHTML2text(
+                        el.attrib.get('data-toc-label', text), self.md, strip_entities=False
+                    ))
                 })
 
                 # Remove the data-toc-label attribute as it is no longer needed


Test patch:
diff --git a/tests/test_extensions.py b/tests/test_extensions.py
index 3ff4f7f..c787886 100644
--- a/tests/test_extensions.py
+++ b/tests/test_extensions.py
@@ -780,6 +780,7 @@ class TestTOC(TestCaseWithAssertStartsWith):
         self.assertStartsWith('<div class="toc">', self.md.toc)
         self.md.reset()
         self.assertEqual(self.md.toc, '')
+        self.assertEqual(self.md.toc_tokens, [])
 
     def testUniqueIds(self):
         """ Test Unique IDs. """
@@ -791,6 +792,21 @@ class TestTOC(TestCaseWithAssertStartsWith):
             '<h1 id="header_1">Header</h1>\n'
             '<h1 id="header_2">Header</h1>'
         )
+        self.assertEqual(
+            self.md.toc,
+            '<div class="toc">\n'
+              '<ul>\n'                                       # noqa
+                '<li><a href="#header">Header</a></li>\n'    # noqa
+                '<li><a href="#header_1">Header</a></li>\n'  # noqa
+                '<li><a href="#header_2">Header</a></li>\n'  # noqa
+              '</ul>\n'                                      # noqa
+            '</div>\n'
+        )
+        self.assertEqual(self.md.toc_tokens, [
+            {'level': 1, 'id': 'header', 'name': 'Header', 'children': []},
+            {'level': 1, 'id': 'header_1', 'name': 'Header', 'children': []},
+            {'level': 1, 'id': 'header_2', 'name': 'Header', 'children': []},
+        ])
 
     def testHtmlEntities(self):
         """ Test Headers with HTML Entities. """
@@ -799,6 +815,17 @@ class TestTOC(TestCaseWithAssertStartsWith):
             self.md.convert(text),
             '<h1 id="foo-bar">Foo &amp; bar</h1>'
         )
+        self.assertEqual(
+            self.md.toc,
+            '<div class="toc">\n'
+              '<ul>\n'                                             # noqa
+                '<li><a href="#foo-bar">Foo &amp; bar</a></li>\n'  # noqa
+              '</ul>\n'                                            # noqa
+            '</div>\n'
+        )
+        self.assertEqual(self.md.toc_tokens, [
+            {'level': 1, 'id': 'foo-bar', 'name': 'Foo &amp; bar', 'children': []},
+        ])
 
     def testRawHtml(self):
         """ Test Headers with raw HTML. """
@@ -807,6 +834,17 @@ class TestTOC(TestCaseWithAssertStartsWith):
             self.md.convert(text),
             '<h1 id="foo-bar-baz">Foo <b>Bar</b> Baz.</h1>'
         )
+        self.assertEqual(
+            self.md.toc,
+            '<div class="toc">\n'
+              '<ul>\n'                                                # noqa
+                '<li><a href="#foo-bar-baz">Foo Bar Baz.</a></li>\n'  # noqa
+              '</ul>\n'                                               # noqa
+            '</div>\n'
+        )
+        self.assertEqual(self.md.toc_tokens, [
+            {'level': 1, 'id': 'foo-bar-baz', 'name': 'Foo Bar Baz.', 'children': []},
+        ])
 
     def testBaseLevel(self):
         """ Test Header Base Level. """
@@ -833,6 +871,12 @@ class TestTOC(TestCaseWithAssertStartsWith):
               '</ul>\n'                                                # noqa
             '</div>\n'
         )
+        self.assertEqual(md.toc_tokens, [
+            {'level': 5, 'id': 'some-header', 'name': 'Some Header', 'children': [
+                {'level': 6, 'id': 'next-level', 'name': 'Next Level', 'children': []},
+                {'level': 6, 'id': 'too-high', 'name': 'Too High', 'children': []},
+            ]},
+        ])
 
     def testHeaderInlineMarkup(self):
         """ Test Headers with inline markup. """
@@ -843,6 +887,18 @@ class TestTOC(TestCaseWithAssertStartsWith):
             '<h1 id="some-header-with-markup">Some <em>Header</em> with '
             '<a href="http://example.com">markup</a>.</h1>'
         )
+        self.assertEqual(
+            self.md.toc,
+            '<div class="toc">\n'
+              '<ul>\n'                                     # noqa
+                '<li><a href="#some-header-with-markup">'  # noqa
+                  'Some Header with markup.</a></li>\n'    # noqa
+              '</ul>\n'                                    # noqa
+            '</div>\n'
+        )
+        self.assertEqual(self.md.toc_tokens, [
+            {'level': 1, 'id': 'some-header-with-markup', 'name': 'Some Header with markup.', 'children': []},
+        ])
 
     def testAnchorLink(self):
         """ Test TOC Anchorlink. """
@@ -942,40 +998,38 @@ class TestTOC(TestCaseWithAssertStartsWith):
     def testWithAttrList(self):
         """ Test TOC with attr_list Extension. """
         md = markdown.Markdown(extensions=['toc', 'attr_list'])
-        text = '# Header 1\n\n## Header 2 { #foo }\n\n## Header 3 { data-toc-label="Foo Bar"}'
+        text = ('# Header 1\n\n'
+                '## Header 2 { #foo }\n\n'
+                '## Header 3 { data-toc-label="Foo Bar"}\n\n'
+                '# Header 4 { data-toc-label="Foo <b>Baz</b>" }')
         self.assertEqual(
             md.convert(text),
             '<h1 id="header-1">Header 1</h1>\n'
             '<h2 id="foo">Header 2</h2>\n'
-            '<h2 id="header-3">Header 3</h2>'
+            '<h2 id="header-3">Header 3</h2>\n'
+            '<h1 id="header-4">Header 4</h1>'
         )
         self.assertEqual(
             md.toc,
             '<div class="toc">\n'
-              '<ul>\n'                                           # noqa
-                '<li><a href="#header-1">Header 1</a>'           # noqa
-                  '<ul>\n'                                       # noqa
-                    '<li><a href="#foo">Header 2</a></li>\n'     # noqa
-                    '<li><a href="#header-3">Foo Bar</a></li>\n' # noqa
-                  '</ul>\n'                                      # noqa
-                '</li>\n'                                        # noqa
-              '</ul>\n'                                          # noqa
+              '<ul>\n'                                            # noqa
+                '<li><a href="#header-1">Header 1</a>'            # noqa
+                  '<ul>\n'                                        # noqa
+                    '<li><a href="#foo">Header 2</a></li>\n'      # noqa
+                    '<li><a href="#header-3">Foo Bar</a></li>\n'  # noqa
+                  '</ul>\n'                                       # noqa
+                '</li>\n'                                         # noqa
+                '<li><a href="#header-4">Foo Baz</a></li>\n'      # noqa
+              '</ul>\n'                                           # noqa
             '</div>\n'
         )
-        self.assertEqual(
-            md.toc_tokens,
-            [
-                {
-                    'level': 1,
-                    'id': 'header-1',
-                    'name': 'Header 1',
-                    'children': [
-                        {'level': 2, 'id': 'foo', 'name': 'Header 2', 'children': []},
-                        {'level': 2, 'id': 'header-3', 'name': 'Foo Bar', 'children': []}
-                    ]
-                }
-            ]
-        )
+        self.assertEqual(md.toc_tokens, [
+            {'level': 1, 'id': 'header-1', 'name': 'Header 1', 'children': [
+                {'level': 2, 'id': 'foo', 'name': 'Header 2', 'children': []},
+                {'level': 2, 'id': 'header-3', 'name': 'Foo Bar', 'children': []}
+            ]},
+            {'level': 1, 'id': 'header-4', 'name': 'Foo Baz', 'children': []},
+        ])
 
     def testUniqueFunc(self):
         """ Test 'unique' function. """