Repository: Python-Markdown/markdown (python)

Issue:
The title from `toc_tokens` ignores the `smarty` extension
Current actual result:
```pycon
>>> import markdown
>>> md = markdown.Markdown(extensions=['toc', 'smarty'])
>>> md.convert('# *Foo* --- `bar`')
'<h1 id="foo-bar"><em>Foo</em> &mdash; <code>bar</code></h1>'
>>> md.toc_tokens
[{'level': 1, 'id': 'foo-bar', 'name': 'Foo --- bar', 'children': []}]
```
Expected result:
<pre>
[{'level': 1, 'id': 'foo-bar', 'name': 'Foo <b>&amp;mdash;</b> bar', 'children': []}]
</pre>

Background where I discovered this: https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1925446505

This happens because `smarty` runs at a lower priority than `toc` (unlike most other treeprocessors) and it doesn't have a chance to kick in.
* https://github.com/Python-Markdown/markdown/blob/a63e6f3416f2e4fc7c2d5edb2768ceed9218df20/markdown/extensions/toc.py#L404
* https://github.com/Python-Markdown/markdown/blob/a63e6f3416f2e4fc7c2d5edb2768ceed9218df20/markdown/extensions/smarty.py#L272

This other extension is presumably affected in the same way, probably among others: 
https://github.com/facelessuser/pymdown-extensions/blob/8f5283f71e8833f1ba0fa0bbe2680e1ad2c6cb19/pymdownx/smartsymbols.py#L167

Something should be done so that all the rest of the treeprocessors are also applied before saving the title. Maybe postprocessors too, rather than applying the unescape functionality out-of-band.

Pull request:
Give smarty treeprocessor higher priority than toc
And add a test for using smarty in ToC headers.

Fixes #1438.

Patch:
diff --git a/docs/changelog.md b/docs/changelog.md
index 884320f..33f05ce 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 * Include `scripts/*.py` in the generated source tarballs (#1430).
 * Ensure lines after heading in loose list are properly detabbed (#1443).
+* Give smarty tree processor higher priority than toc (#1440).
 
 ## [3.5.2] -- 2024-01-10
 
diff --git a/markdown/extensions/smarty.py b/markdown/extensions/smarty.py
index 0ce7772..7a7c952 100644
--- a/markdown/extensions/smarty.py
+++ b/markdown/extensions/smarty.py
@@ -269,7 +269,7 @@ class SmartyExtension(Extension):
             self.educateDashes(md)
         inlineProcessor = InlineProcessor(md)
         inlineProcessor.inlinePatterns = self.inlinePatterns
-        md.treeprocessors.register(inlineProcessor, 'smarty', 2)
+        md.treeprocessors.register(inlineProcessor, 'smarty', 6)
         md.ESCAPED_CHARS.extend(['"', "'"])
 
 


Test patch:
diff --git a/tests/test_syntax/extensions/test_smarty.py b/tests/test_syntax/extensions/test_smarty.py
index aeb0988..8a17674 100644
--- a/tests/test_syntax/extensions/test_smarty.py
+++ b/tests/test_syntax/extensions/test_smarty.py
@@ -198,3 +198,26 @@ class TestSmartyCustomSubstitutions(TestCase):
             'Must not be confused with &sbquo;ndash&lsquo;  (\u2013) \u2026 ]</p>'
         )
         self.assertMarkdownRenders(text, html)
+
+
+class TestSmartyAndToc(TestCase):
+
+    default_kwargs = {
+        'extensions': ['smarty', 'toc'],
+    }
+
+    def test_smarty_and_toc(self):
+        self.assertMarkdownRenders(
+            '# *Foo* --- `bar`',
+            '<h1 id="foo-bar"><em>Foo</em> &mdash; <code>bar</code></h1>',
+            expected_attrs={
+                'toc_tokens': [
+                    {
+                        'level': 1,
+                        'id': 'foo-bar',
+                        'name': 'Foo &mdash; bar',
+                        'children': [],
+                    },
+                ],
+            },
+        )