Repository: Python-Markdown/markdown
Pull request #700 (2018-08-17 17:53:14)

Support custom labels in TOC.
When the Table of Contents Extension is used in conjunction with the Attribute Lists Extension  and a `data-toc-label` [attribute][1] is defined on a header, the content of the  `data-toc-label` attribute is now used as the content of the Table of Contents item for that header.

[1]: https://developer.mozilla.org/en-US/docs/Learn/HTML/Howto/Use_data_attributes

facelessuser: Interesting workaround for sane tocs with complex headers.
waylan: Yeah, I've had the idea bouncing around for a while, but was thinking of using the `title` attribute. I wasn't crazy about it though as there is a legitimate use for titles and they are potentially visible to end users (and screen readers). Today it occurred to me that we could use a `data-*` attribute. And then I had it working with only one line of code. Writing the documentation was the hardest part.

We should probably still properly fix #660, although this would serve as a workaround in the meantime. Note sure if there is anything else we could do for #699 though.
waylan: The last commit (537f418) just added a another (unrelated) feature to TOC:

A new `toc_tokens` attribute has been added to the Markdown class, which contains the raw tokens used to build the Table of Contents. Users can use this to build their own custom Table of Contents rather than needing to parse the HTML available on the `toc` attribute of the Markdown class.

Patch:
diff --git a/.gitignore b/.gitignore
index 96c6778..ef7b434 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,18 +1,70 @@
-.DS*
-*.pyc
-*.bak
-*.swp
-test-output.html
-build/*
-dist/*
-tmp/*
-site/*
-MANIFEST
-.venv
-*~
-*#*
-.tox/*
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
 .coverage
-htmlcov/*
-*komodo*
-Markdown.egg-info/*
\ No newline at end of file
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+test-output.html
+
+# Translations
+*.mo
+*.pot
+
+# Scrapy stuff:
+.scrapy
+
+# PyBuilder
+target/
+
+# IPython Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# virtualenv
+venv/
+ENV/
+
+# MkDocs documentation
+site/
diff --git a/docs/change_log/release-3.0.md b/docs/change_log/release-3.0.md
index 85a5d27..ab6b83e 100644
--- a/docs/change_log/release-3.0.md
+++ b/docs/change_log/release-3.0.md
@@ -202,5 +202,17 @@ The following new features have been included in the release:
 * A new `toc_depth` parameter has been added to the
   [Table of Contents Extension](../extensions/toc.md).
 
+* A new `toc_tokens` attribute has been added to the Markdown class by the
+  [Table of Contents Extension](../extensions/toc.md), which contains the raw
+  tokens used to build the Table of Contents. Users can use this to build their
+  own custom Table of Contents rather than needing to parse the HTML available
+  on the `toc` attribute of the Markdown class.
+
+* When the [Table of Contents Extension](../extensions/toc.md) is used in
+  conjunction with the [Attribute Lists Extension](../extensions/attr_list.md)
+  and a `data-toc-label` attribute is defined on a header, the content of the
+  `data-toc-label` attribute is now used as the content of the Table of Contents
+  item for that header.
+
 * Additional CSS class names can be appended to
   [Admonitions](../extensions/admonition.md).
diff --git a/docs/extensions/toc.md b/docs/extensions/toc.md
index 5038428..f6111b2 100644
--- a/docs/extensions/toc.md
+++ b/docs/extensions/toc.md
@@ -56,7 +56,7 @@ would generate the following output:
   </ul>
 </div>
 <h1 id="header-1">Header 1</h1>
-<h1 id="header-2">Header 2</h1>
+<h2 id="header-2">Header 2</h2>
 ```
 
 Regardless of whether a `marker` is found in the document (or disabled), the
@@ -70,6 +70,67 @@ template. For example:
 >>> page = render_some_template(context={'body': html, 'toc': md.toc})
 ```
 
+The `toc_tokens` attribute is also available on the Markdown class and contains
+a nested list of dict objects. For example, the above document would result in
+the following object at `md.toc_tokens`:
+
+```python
+[
+    {
+        'level': 1,
+        'id': 'header-1',
+        'name': 'Header 1',
+        'children': [
+            {'level': 2, 'id': 'header-2', 'name': 'Header 2', 'children':[]}
+        ]
+    }
+]
+```
+
+Note that the `level` refers to the `hn` level. In other words, `<h1>` is level
+`1` and `<h2>` is level `2`, etc. Be aware that improperly nested levels in the
+input may result in odd nesting of the output.
+
+### Custom Labels
+
+In most cases, the text label in the Table of Contents should match the text of
+the header. However, occasionally that is not desirable. In that case, if this
+extension is used in conjunction with the [Attribute Lists Extension] and a
+`data-toc-label` attribute is defined on the header, then the contents of that
+attribute will be used as the text label for the item in the Table of Contents.
+For example, the following Markdown:
+
+[Attribute Lists Extension]: attr_list.md
+
+```md
+[TOC]
+
+# Functions
+
+## `markdown.markdown(text [, **kwargs])` { #markdown data-toc-label='markdown.markdown' }
+```
+would generate the following output:
+
+```html
+<div class="toc">
+  <ul>
+    <li><a href="#functions">Functions</a></li>
+      <ul>
+        <li><a href="#markdown">markdown.markdown</a></li>
+      </ul>
+  </ul>
+</div>
+<h1 id="functions">Functions</h1>
+<h2 id="markdown"><code>markdown.markdown(text [, **kwargs])</code></h2>
+```
+
+Notice that the text in the Table of Contents is much cleaner and easier to read
+in the context of a Table of Contents. The `data-toc-label` is not included in
+the HTML header element. Also note that the ID was manually defined in the
+attribute list to provide a cleaner URL when linking to the header. If the ID is
+not manually defined, it is always derived from the text of the header, never
+from the `data-toc-label` attribute.
+
 Usage
 -----
 
diff --git a/markdown/extensions/toc.py b/markdown/extensions/toc.py
index 2741037..f6121c2 100644
--- a/markdown/extensions/toc.py
+++ b/markdown/extensions/toc.py
@@ -247,15 +247,20 @@ class TocTreeprocessor(Treeprocessor):
                 toc_tokens.append({
                     'level': int(el.tag[-1]),
                     'id': el.attrib["id"],
-                    'name': text
+                    'name': el.attrib.get('data-toc-label', text)
                 })
 
+                # Remove the data-toc-label attribute as it is no longer needed
+                if 'data-toc-label' in el.attrib:
+                    del el.attrib['data-toc-label']
+
                 if self.use_anchors:
                     self.add_anchor(el, el.attrib["id"])
                 if self.use_permalinks:
                     self.add_permalink(el, el.attrib["id"])
 
-        div = self.build_toc_div(nest_toc_tokens(toc_tokens))
+        toc_tokens = nest_toc_tokens(toc_tokens)
+        div = self.build_toc_div(toc_tokens)
         if self.marker:
             self.replace_marker(doc, div)
 
@@ -263,6 +268,7 @@ class TocTreeprocessor(Treeprocessor):
         toc = self.md.serializer(div)
         for pp in self.md.postprocessors:
             toc = pp.run(toc)
+        self.md.toc_tokens = toc_tokens
         self.md.toc = toc
 
 
@@ -310,6 +316,7 @@ class TocExtension(Extension):
 
     def reset(self):
         self.md.toc = ''
+        self.md.toc_tokens = []
 
 
 def makeExtension(**kwargs):  # pragma: no cover


Test patch:
diff --git a/tests/test_extensions.py b/tests/test_extensions.py
index 532ed6f..5489887 100644
--- a/tests/test_extensions.py
+++ b/tests/test_extensions.py
@@ -852,24 +852,40 @@ class TestTOC(TestCaseWithAssertStartsWith):
     def testWithAttrList(self):
         """ Test TOC with attr_list Extension. """
         md = markdown.Markdown(extensions=['toc', 'attr_list'])
-        text = '# Header 1\n\n## Header 2 { #foo }'
+        text = '# Header 1\n\n## Header 2 { #foo }\n\n## Header 3 { data-toc-label="Foo Bar"}'
         self.assertEqual(
             md.convert(text),
             '<h1 id="header-1">Header 1</h1>\n'
-            '<h2 id="foo">Header 2</h2>'
+            '<h2 id="foo">Header 2</h2>\n'
+            '<h2 id="header-3">Header 3</h2>'
         )
         self.assertEqual(
             md.toc,
             '<div class="toc">\n'
-              '<ul>\n'                                        # noqa
-                '<li><a href="#header-1">Header 1</a>'        # noqa
-                  '<ul>\n'                                    # noqa
-                    '<li><a href="#foo">Header 2</a></li>\n'  # noqa
-                  '</ul>\n'                                   # noqa
-                '</li>\n'                                     # noqa
-              '</ul>\n'                                       # noqa
+              '<ul>\n'                                           # noqa
+                '<li><a href="#header-1">Header 1</a>'           # noqa
+                  '<ul>\n'                                       # noqa
+                    '<li><a href="#foo">Header 2</a></li>\n'     # noqa
+                    '<li><a href="#header-3">Foo Bar</a></li>\n' # noqa
+                  '</ul>\n'                                      # noqa
+                '</li>\n'                                        # noqa
+              '</ul>\n'                                          # noqa
             '</div>\n'
         )
+        self.assertEqual(
+            md.toc_tokens,
+            [
+                {
+                    'level': 1,
+                    'id': 'header-1',
+                    'name': 'Header 1',
+                    'children': [
+                        {'level': 2, 'id': 'foo', 'name': 'Header 2', 'children': []},
+                        {'level': 2, 'id': 'header-3', 'name': 'Foo Bar', 'children': []}
+                    ]
+                }
+            ]
+        )
 
     def testUniqueFunc(self):
         """ Test 'unique' function. """