Repository: mkdocs/mkdocs

Issue:
Header and navigation sidebars display incorrectly arbitrary markup
Hello!

## Since when does the bug occur?

I recently upgraded my `mkdocs` package from `1.4.2` to `1.5.2`, and I noticed this bug when...
* I don't specify a page title in `mkdocs.yml` file or in the page metadata (in yaml).
* I use an emoji (HTML in general) in the first level section title in the markdown file:

```md
# Intégration et Déploiement Continu sur :simple-gitlab: Gitlab
```

For example (with `mkdocs-material` theme and `readthedocs` theme):

![263000285-555a71e9-3530-4a63-906a-17617c1d0be3](https://github.com/mkdocs/mkdocs/assets/46783975/fafe7715-678b-4813-9535-c6ddf3df9acd)
![263459179-2afb6262-bde4-4727-b0e3-c5c4da035873](https://github.com/mkdocs/mkdocs/assets/46783975/90edf72c-61df-4d96-a8ba-aed2e35332a0)

... while it was displayed like that before:

![262992157-dcebad1a-82b1-445d-b250-8ef7aa34bafb](https://github.com/mkdocs/mkdocs/assets/46783975/8e25e34a-036d-4358-8929-f08d473e6004)

More generally, some parts like the search page of `readthedocs` theme don't handle well the HTML in title sections.

## External links:

[`readthedocs` theme pictures and bug explanations](https://github.com/squidfunk/mkdocs-material/issues/5893) are from @squidfunk.


Discussion:
@oprypin friendly ping, since this is a pretty serious bug. As of MkDocs 1.5, the page title contains HTML, which does not only break our project Material for MkDocs, but other themes like ReadTheDocs, and potentially many plugins that are not prepared to handle HTML in `page.title`. It's new behavior that was introduced in MkDocs 1.5.0 and effectively prohibits to use markup in headlines, as this markup doesn't seem to be stripped from the page title anymore. It is now also part of the search index, for which the search that comes with MkDocs is not prepared:

![263459257-b8f7105a-3f4f-4536-a472-b2342b625b6c](https://github.com/mkdocs/mkdocs/assets/932156/accf9c91-b0a6-46b0-93f0-1a9e3b3d7ee2)

The changelog of MkDocs 1.5.0 only talks about [a new method for deducing the page title](https://www.mkdocs.org/about/release-notes/#page-titles-will-be-deduced-from-any-style-of-heading), but not about this change in behavior which, in my opinion, is a breaking change that requires the ecosystem to adapt. For this reason, I assume it's an unintended bug that we need to fix.
Just to make sure I get it right: what would be the expected behavior here? Silently ignore any `<span>` containing the rendered SVG of the emoji? Raise an error and crash the build?
The behavior of MkDocs 1.4 – filter HTML tags from page titles.

__Edit__: If that is the new desired behavior, it should go into the docs and it should actually be a breaking release (= MkDocs 2.0) because _all_ plugins that process `page.title` are impacted and probably need to be adjusted.
Possibly related: https://github.com/waylan/mkdocs-nature/pull/5. Only recently did we introduce HTML `<code>` tags in some parts of the navigation of Python-Markdown's docs (API reference, see https://github.com/Python-Markdown/markdown/pull/1379) that were then shown in the browser's tab title (and therefore window manager's window title). Maybe the change wouldn't have been needed with MkDocs 1.4. Will check and report back.

UPDATE: well nope, `<code>` tags weren't stripped with MkDocs 1.4.3 either. Though the page titles here are obtained from the `nav` directly (well, through literate-nav) rather than Markdown page H1 titles.

So IIUC, the behavior differs whether a title is given in `nav` or in the page? It would be nice to reconcile both cases. Maybe that's what MkDocs 1.5 did, not stripping anything in any case.
Maybe we need to modify the python-markdown preprocessing for the title parser.

Gentle ping to @facelessuser who I suspect is a master in python-markdown processors. It's quite obscure to me I'm afraid so far...

Could we tweak the [title processor](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/structure/pages.py#L268) to ignore :emoji: formatting?
@ultrabug I'm not sure I understand the question. Can you break this down for me? 
MkDocs 1.5 have a [title processor](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/structure/pages.py#L268) which is now in charge of rendering the title of the page using a custom python-markdown `Treeprocessor`.

Emojis in page title which are rendered as SVG (for example) do break the page structure and CSS as demonstrated here.

I was wondering if we could alter our custom `Treeprocessor` for title abstraction to make it **NOT** tokenize the `:(.*):` emoji syntax maybe?
It might be an issue of ordering – the title processor will run before emoji is being processed, and then after the title processor removed all markup, emojis keeps adding stuff to it. That's at least my theory.
Emoji is an inline processor. These get run before tree processors.

1. Preprocessor
2. Block Processor
3. Inline Processor
4. Tree Processor
5. Post Processor
Python Markdown allows you to specify special labels if you need to control exactly what is used for TOC etc.: https://python-markdown.github.io/extensions/toc/#custom-labels
If you are hoping for TOC to accept some way to exclude specific syntax, you will probably be disappointed. It should not have knowledge of specific (especially 3rd party) extensions. I'm sure you can imagine the maintenance nightmare that would be as other extensions popup that also want to be excluded.

Help me out, what exactly is the end result you want? What is your expectation of how things should work and why would it solve your problem? If I am to be of any help, I need to understand the end goal. Maybe there is a creative way to workaround this. Maybe one I could implement into emoji (if necessary). I just need to understand what you are hoping to accomplish.
The endgoal is to sanitize the `page.title` of any HTML tags. This is not done anymore due to some bug in the new title extraction processor. I'm not sure we need to change something in the emoji extension for that – it worked before, so it's likely that we need to fix the title tree processor.
I've investigated and think I know what the problem is:

https://github.com/mkdocs/mkdocs/blob/f94ab3f62d0416d484d81a0c695c8ca86ab3b975/mkdocs/structure/pages.py#L439-L448

1. Anchors links are removed to omit "Permalink" being part of the extracted title
2. All tags are stripped by just extracting the text with `itertext()`
3. Stashed content is reinjected, which includes the `<svg>` element

A brute force approach would be to strip all tags again, but maybe there's a better solution.
First off let's make a reproduction case and clarify the behavior of markup in nav headers.



mkdocs.yml:
```yaml
site_name: My Docs

markdown_extensions:
  - pymdownx.emoji:
      emoji_index: !!python/name:materialx.emoji.twemoji
      emoji_generator: !!python/name:materialx.emoji.to_svg
```



docs/index.md:
```markdown
# Title *title* <i>title</i> :material-book:
```



There is no change in the content itself but there is change in the nav rendering. Each theme behaves consistently with each other.

---

<table><tr><th>MkDocs 1.4.3</th><th>MkDocs 1.5.2</th></tr>
<tr><td>

> **Title \*title\* *title* :material-book:**

Markdown is ignored/preserved,  
raw HTML is applied

</td><td>

> **Title title *title* :book:**

Markdown is applied and stripped but  
stashed Markdown is applied, raw HTML is applied

</td></tr><tr><td>

"mkdocs" theme
![image](https://github.com/mkdocs/mkdocs/assets/371383/0ca13ff2-adf1-4662-9c93-866ba53ad4b6)

</td><td>

"mkdocs" theme
![image](https://github.com/mkdocs/mkdocs/assets/371383/69683b00-c4c5-4f73-8996-b8b063a432b5)

</td></tr><tr><td>

"readthedocs" theme
![image](https://github.com/mkdocs/mkdocs/assets/371383/d522ad88-1e12-45e8-9107-8ab11e8b9f55)

</td><td>

"readthedocs" theme
![image](https://github.com/mkdocs/mkdocs/assets/371383/09776f6e-c6e8-491e-8bfb-1aaab6f87308)

</td></tr><tr><td>

"material" theme - 9.1.20
![image](https://github.com/mkdocs/mkdocs/assets/371383/a729aa47-1911-4ec9-b319-9a96fbf8f932)

</td><td>

"material" theme - 9.1.20
![image](https://github.com/mkdocs/mkdocs/assets/371383/252b5c5c-e4e9-43e0-8a55-b2181d8e57ff)

</td></tr></table>

All in all both behaviors are wildly inconsistent. If I had to pick one, probably the one in MkDocs 1.5 makes more sense, but that doesn't help much.

Emojis were producing poor results both before and after, just differently.
Only "material" theme had somewhat proper styling for the emoji.

---

And there is the repro case from the issue on mkdocs-material, indeed the behavior is somehow different.
But I haven't looked why exactly the theme behaves like that - at first glance seems theme-specific.

<table><tr><th>MkDocs 1.4.3</th><th>MkDocs 1.5.2</th></tr>
<tr><td>

"material" theme - 9.1.20-insiders-4.37.0
![image](https://github.com/mkdocs/mkdocs/assets/371383/3f366d1d-3380-4afa-ae1e-de66685127a0)

</td><td>

![image](https://github.com/mkdocs/mkdocs/assets/371383/de42db14-95f4-42d8-87ac-44cbe8d0d3be)


</td></tr></table>

---

There's also the separate matter that there is a "material/typeset" plugin (not public) that was relying on the behavior that Markdown is being ignored/preserved as it's being moved to the nav title, and then re-applying that Markdown in its own way. So emojis were fully supported only on mkdocs-material-insiders in its own way, and this change made that not function anymore.

---

And it is *definitely* a huge issue that the HTML now reaches directly into search results

![image](https://github.com/mkdocs/mkdocs/assets/371383/88478d13-6ef4-4603-8ed1-16a98a6b82ff)


So some things to note:

* Even in old MkDocs, HTML was able to get through. It wasn't actually sanitized, it's just that Markdown wasn't rendered, adding more of its own HTML.

* If MkDocs now actually starts to sanitize all HTML tags, then emojis will be completely obliterated.
Regarding why the current behavior is like that, I'm quite sure that @squidfunk has it spot on - https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1719611003

But it is unclear what the actual best outcome would be, seeing as, again, actually stripping all HTML would just make emojis actually impossible to support in titles. Whereas in the current state, to support them, only some CSS fixes would be needed.
And yet another aspect - indeed, specifying the page title in the `nav` YAML config still has exactly the same behavior as with MkDocs 1.4 (ignoring/preserving Markdown) - it's only the titles obtained from the page content that now selectively render Markdown, which is certainly unfortunate.
https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1719308655
Indeed another huge deal - similarly to the search results - the HTML also ends up propagating to the `<title>` tag in both built-in themes, so themes need to manually call `striptags`, or otherwise the browser's title bar can get a large chunk of HTML.

**But this was always partly the case** if one were to write raw HTML for the page title, just that now Markdown can also insert a huge SVG emoji and it will end up in the page title as raw markup
One pretty straightforward resolution here would be to indeed almost totally rollback to pre-1.5 behavior, even if it's not ideal. There's actually a way to do that that doesn't require rolling back the new title detection.

* Created #3394 as such
Always amazed by your thoroughness @oprypin, thank you for doing the heavy lifting of investigating the whys and hows.
Hmm. Can't we change the priority of Markdown extensions, so that the stashes are restored before the tags are stripped? IMHO, this would be a better solution, but I'm probably missing something why this is not possible.
> There's also the separate matter that there is a "material/typeset" plugin (not public) that was relying on the behavior that Markdown is being ignored/preserved as it's being moved to the nav title, and then re-applying that Markdown in its own way. So emojis were fully supported only on mkdocs-material-insiders in its own way, and this change made that not function anymore.

The typeset plugin is actually an attempt to allow for a controlled variant of what's reported in this issue, particularly that themes can opt into allowing for rich text in _specific_ parts without forcing HTML tags down every theme and plugin. Quite a few popular plugins in the ecosystem would need to be adapted if we would opt for MkDocs 1.5 behavior, which is why I consider this to be a breaking change that mandates a major release (if not declared a bug).
> Hmm. Can't we change the priority of Markdown extensions, so that the stashes are restored before the tags are stripped? IMHO, this would be a better solution, but I'm probably missing something why this is not possible.

@squidfunk I'm just quite lost regarding *what the output should be conceptually*. In terms of how to implement it, anything is doable.
The most helpful feedback that I'd like to see is what value of `title` you expect to see for a given input Markdown.

E.g.

* Input: <code>\\*Hello --- \*beautiful\* \`world\`</code>
  MkDocs 1.4: <code>\\*Hello --- \*beautiful\* \`world\`</code>
  MkDocs 1.5: <code>*Hello &mdash; beautiful world</code>

* Input: `Title *title* <i>title</i> :material-book:`
  MkDocs 1.4: `Title *title* <i>title</i> :material-book:`
  MkDocs 1.5: `Title title <i>title</i> <svg blob...></svg>`

> The most helpful feedback that I'd like to see is what value of title you expect to see for a given input Markdown.

Ideally a title without Markdown formatting and HTML tags. If that's not possible, MkDocs 1.4 behavior.
Thanks for clarifying things better @oprypin 

I allowed myself to illustrate the effects of [the proposed PR](https://github.com/mkdocs/mkdocs/pull/3394) to link it to this discussion
As @pawamoy [mentioned](https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1719308655) above, we encountered a similar issue and addressed it in waylan/mkdocs-nature#5. In our case the HTML tags were coming from connect generated from a script so we were not actually affected by this exact issue (in that the source our page titles were not parsed Markdown). Regardless, we had HTML tags in our page titles which were being inserted in our templates in places they shouldn't be.

The solution we used was to add the `striptags` filter to our templates where appropriate. In fact, in some places, we wanted the HTML tags, but in other places they were not desirable. Obviously, we want no HTML tags in the `<title>` tag for a page. Same goes for `title` attributes. But we do want the HTML tags in link labels (for example in the nav). By having MkDocs leave the tags in, we have total control in our templates. Consider this pseudo example:

```html
<a href="{{ page.url }}" title="{{ page.title|striptags }}">{{ page.title }}</a>
```

If the current behavior is reversed, then all titles all of the time would never contain any HTML tags. In our case, we wanted `<code>` tags to wrap each page title as each represents a Python module. If all of those `<code>` tags were stripped, our spellchecking CI test would throw a fit and many of the module names are not in the dictionary and there would be no way to work around it.

I realize that a breaking change was introduced to MkDocs here and now there are compatibility issues for previous releases of themes. So, leaving the behavior as-is is probably not a reasonable way forward. However, from my perspective, the current behavior is actually preferred for my specific use case. Just throwing that out there as a consideration.

As an aside, it occurred to me that a future update could reverse MkDocs behavior and now we would have unnecessary calls to the `striptags` filter in our templates. But I am okay with that as it ensures we never get tags were they are not supposed to be regardless of any future updates. It might be good practice for all theme developers to be using a similar approach with their templates.

Finally, I will note that any future reversal could have no effect on our issue as we are not getting our titles from Markdown text. If so, that is good for us as we can continue to use our generated titles with HTML tags.
> * Input: `Title *title* <i>title</i> :material-book:`
>   MkDocs 1.4: `Title *title* <i>title</i> :material-book:`
>   MkDocs 1.5: `Title title <i>title</i> <svg blob...></svg>`

I would say the last one would be most desirable of the choices provided. That can be used anywhere. Were we want HTML we can use it as-is. Where we don't want any HTML tags, then we can use the `striptags` filter, which results in `Title title title`. That seems much preferred to `Title *title* <i>title</i> :material-book:`, which everyone seems to be asking for here.

Of course, that ignores the many existing plugins and themes which have been expecting `Title *title* <i>title</i> :material-book:` for some time. I do not see a clear path to this being a graceful change.
This issue is quite stuck.

* I'm probably not going to move forward with https://github.com/mkdocs/mkdocs/pull/3394
   -the old behavior really is a lot more wrong than the current one

Any feedback would still be very helpful.
Is there any potential solution to this? I'm unfamiliar with the internals.
I second @ofek – it would be great if this could be sorted out. IMHO, it is a pretty severe bug that causes a lot of frustration among users and more work for maintainers, answering issues and discussions of users that run into this.

IMHO, there should either be two variants of `Page.title`, one that is stripped of all markup, and one that contains HTML, or at least no markup in `Page.title` ever. This would allow to use the title inside contexts that don't support markup, as well inside the navigation or templates. How can we make this happen quicker to reduce frustration?
I agree with @oprypin that reverting does not make sense.

> IMHO, there should either be two variants of `Page.title`, one that is stripped of all markup, and one that contains HTML

@squidfunk I guess I don't understand the problem as this is already possible. Just use the `striptags` filter in your templates were markup does not belong. Why doesn't this resolve the issue for you? 

```html
<a href="{{ page.url }}" title="{{ page.title|striptags }}">{{ page.title }}</a>
```
> @squidfunk I guess I don't understand the problem as this is already possible. Just use the striptags filter in your templates were markup does not belong. Why doesn't this resolve the issue for you?

As I wrote in https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1876260518:

> This would allow to use the title inside contexts that don't support markup, as well inside the navigation or templates.

You're right when it comes to templates – definitely a feasible solution, which, however, mandates that all template authors change all occurrences. I have no problem of doing that if it would solve all problems that appeared since 1.5, but it doesn't: even more problematic is the consumption of `Page.title` in plugins, e.g., MkDocs own search plugin as I reported in https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1718865058. As an example, search for "config" on [mkdocs.org](https://www.mkdocs.org/) and scroll down:

<img width="792" alt="Bildschirm­foto 2024-01-09 um 08 37 56" src="https://github.com/mkdocs/mkdocs/assets/932156/722ca9d2-bc1e-45ca-9ef1-f54eee890334">

`striptags` is not the solution for plugin authors.
> `striptags` is not the solution for plugin authors.

Thanks, that clears up what I was missing. I've been away for too long...

> You're right when it comes to templates – definitely a feasible solution, which, however, mandates that all template authors change all occurrences.

This is the tricky part. I would consider the behavior prior to 1.5 to be a bug. Therefore the existing change is a fix. Of course, the problem is that the fix requires all users to also make a change. So, is there a way to provide a fix which does not have that requirement?

Such a solution would need to detect whether it was in a context which expected markup or not. I don't know what that would look like. `page.title` could be a callable which accepted an argument (`page.title(striptags=False)`) but that would require every call (not just the no-markup calls) to be changed. 

In fact your proposal doesn't address that concern either. If we had 2 variables, say `page.title` and `page.title_text`. Every call to `page.title` which should not receive markup will need to be changed to `page.title_text`. Those instances can just as easily be changed to `striptags(page.title)` (in plugins) or `{{ page.title|striptags }}` (in templates). Technically, plugins don't even need to wait for MkDocs to provide such a function, as they could implement their own.

My point is that there is no way to do a graceful deprecation here, unless a completely new variable was introduced. I suppose `page.title` could be reverted to the old behavior and then `page.title_html` and `page.title_text` (or whatever they are called) could get the new behavior. In that way, existing themes and plugins could continue to work as before until they are updated for the new API.

My resistance to that is that `page.title` is a much better name in my opinion. Although, upon further reflection, it occurs to me that we could make a `Title` class which contains the raw title, and provides properties which return the various different formats. The `__str__` method could return the old behavior at first with a deprecation warning and then with some future release change to the new behavior. But `page.title.html` would always return the HTML version and `page.title.text` would always return the plan text version of the title. Maybe even `page.title.src` could return the unaltered source text.

The problem with the last 2 suggestions is that we already have had 1.5 out in the wild for a while now and any new work will be expecting the existing behavior. To revert `page.title` now would be a regression. I know I personally would need to go change all my work over the last few months, which I don't want to do. This would have been the sort of thing to work out when the change was first introduced.
> This is the tricky part. I would consider the behavior prior to 1.5 to be a bug. Therefore the existing change is a fix. Of course, the problem is that the fix requires all users to also make a change. 

By that logic, even though you consider it a bug, it was a breaking change mandating all template authors to change their templates, which, IMHO, would require a major release, so 1.5 should have been 2.0.

> In fact your proposal doesn't address that concern either. If we had 2 variables, say page.title and page.title_text. Every call to page.title which should not receive markup will need to be changed to page.title_text

IMHO, `Page.title` should _never_ receive markup – exactly what was and is expected by all plugins prior to 1.5. MkDocs has a history of keeping backward compatibility, e.g., the search plugin being auto-enabled but disappearing once `plugins` is defined (which leads to a lot of maintenance and clarification cost down the line), so I regard this to be similar.

> Those instances can just as easily be changed to striptags(page.title) (in plugins) or {{ page.title|striptags }} (in templates).

Well, we would have to change hundreds of plugins and at least dozens of templates, if you take a look at the [catalog](https://github.com/mkdocs/catalog), so I wouldn't classify this as an easy change, but a _very costly_ change.

> My point is that there is no way to do a graceful deprecation here, unless a completely new variable was introduced.

IMHO, a new variable is the only way forward, keeping backward compatibility, keeping cost of changes down and separating concerns (that's what it is) – cases where no markup is expected (strict), and cases where markup is wanted (loose).

> we could make a Title class which contains the raw title, and provides properties which return the various different formats. The __str__ method could return the old behavior at first with a deprecation warning and then with some future release change to the new behavior. But page.title.html would always return the HTML version and page.title.text

That sounds good to me! __str__ should return the plain string and `.html` or `.content` to be coherent ith `page.content` should return markup. IMHO, no deprecation warning should be added. `Page.title` should call `Title.__str__` – I'd regard this as a very good and convenient API.

> The problem with the last 2 suggestions is that we already have had 1.5 out in the wild for a while now and any new work will be expecting the existing behavior. To revert page.title now would be a regression. 

Yes and no. 1.5 introduced a bug in `Page.title`, so we would fix a regression, which is perfectly fine, IMHO. However, I don't think anybody relies on the current behavior which is largely undefined and incoherent. Thus, I'd declare it as a bug. The number of plugins and hooks that rely on this new behavior should be very small. Disclaimer: this is my gut feeling, I have no data to back that up.

> I know I personally would need to go change all my work over the last few months, which I don't want to do.

I don't understand. You're only stripping tags, so if there are no tags, you're fine. If `Title.__str__` returns the title without markup, no changes are required, at least as I can see it.

> This would have been the sort of thing to work out when the change was first introduced.

That ship is sailed, but we really need to limit the damage this is doing.
> By that logic, even though you consider it a bug, it was a breaking change mandating all template authors to change their templates, which, IMHO, would require a major release, so 1.5 should have been 2.0.

I agree, but as you acknowledge, we can't go back in time. So let's focus on what to do going forward.

> IMHO, `Page.title` should _never_ receive markup – exactly what was and is expected by all plugins prior to 1.5.

I'm not sure I agree with that completely. In fact, there are occasions when markup is wanted and desired for a page title. However, there are also occasions when it is definitely not desired. Unfortunately, MkDocs has always only ever accommodated one or the other. The problem was that the switch happened suddenly and without warning.

For me, the hesitation in reverting is that it has become more common in recent years for markup other than basic inline Markdown to be found in page titles (see some of the examples given in this issue). How should that be rendered in a page title? There are also differing opinions about that. Some seem to be asking for the text to be left as unrendered markup, whereas others are asking for a full render and then extract the plain text from the render. 

@oprypin and I are arguing to not revert to the previous behavior because we fall into the later camp. We don't think the unrendered markup should be passed through to the plain text title. The only way to do that is to first do a full render, which requires the existing solution with a call to `striptags` (or similar) afterward. In other words, the "fix" is to add a `striptags` call, not to revert the code.

> > I know I personally would need to go change all my work over the last few months, which I don't want to do.
> 
> I don't understand. You're only stripping tags, so if there are no tags, you're fine. If `Title.__str__` returns the title without markup, no changes are required, at least as I can see it.

Some of my recent work has included titles being dynamically inserted into the body of pages and those titles include markup which I want preserved and rendered (specifically I needed code spans to be preserved). If `page.title` was plain text only, then I would need to go through all that code and retrieve the titles differently so that I could get the title with markup preserved.

So, how do we allow my use case but at the same time also provide a way to get the plain text? I answered that with multiple proposals in my last comment. The next question is what should be the default? Obviously, based on my stated use case, I would prefer 1.5's behavior to be the default as it requires less work for me personally. But...

> we would have to change hundreds of plugins and at least dozens of templates, if you take a look at the [catalog](https://github.com/mkdocs/catalog), so I wouldn't classify this as an easy change, but a _very costly_ change.

I agree. Unfortunately, this is where the core developers can get a blind spot. I wasn't involved in the change in 1.5 and didn't learn about it until my first comment on this issue, so I can only guess on how this got missed. But I also understand the resistance to revert for the reasons stated above.

In the end, I think the `Title` class proposal is probably the best way forward. But I'm not the one making the final call anymore. And I doubt I would find the time to implement it myself anytime soon.

Regardless of what happens here (a `striptags` call is added internally, code is reverted, or something else), I would argue that theme and plugin developers should add `striptag` (or similar) calls to anything that is in a context in which no markup should be included. That is just good practice and avoids the need to make changes in the future when upstream changes happen, whether those upstream changes are intentional or not.
> Regardless of what happens here (a `striptags` call is added internally, code is reverted, or something else), I would argue that theme and plugin developers should add `striptag` (or similar) calls to anything that is in a context in which no markup should be included. That is just good practice and avoids the need to make changes in the future when upstream changes happen, whether those upstream changes are intentional or not.

To illustrate, lets assume my `Title` class proposal gets adopted such that `page.title` and `page.title.text` both return the same plain text. If a template was to reference the later, which is explicitly plain text, then I would not expect a need to pass that through a `striptags` filter. However if using `page.title`, which is only implicitly plain text, I would expect it to always be passed through the `striptags` filter (just as a safety precaution, even if unnecessary). Obviously, `page.title.html` would not be passed through the filter because we can reasonably assume one would not call that in a context were markup is not appropriate.

This is all to say that regardless of what happens here, I think that all theme and plugin devs should audit their projects and make edits to always be explicit about striping out markup where that is desired. I have done that to my projects and this is now a non-issue for me. Of course, I'm not maintaining the most popular MkDocs theme which explicitly supports many various plugins (and implicitly supports many more), some of which are no longer being actively maintained. I can only imagine how frustrating that would be.
Okay, @waylan, I think we will not reach a common understanding of what's the best way forward here. I've invested quite some time into arguing why this should be fixed quickly, and how a fix that has _the least cost of change for the entire ecosystem_ could look like. You made some very good proposals with the `Title` class, but it seems we don't agree where this change should be applied. You're effectively offloading it to the ecosystem instead of fixing the root cause.

Additionally, `striptags` is a bandaid, as you would always need to apply it retroactively at the location of use. This also means potential additional runtime cost due to reparsing strings, which, when building a few thousand sites, might add up and slow down builds. It is just not how I, _personally_, work. I always try to fix problems at the root, if that is possible, not telling my users to apply plasters basically everywhere.

I have nothing more valuable to contribute to this conversation, as I'm already very busy with keeping Material for MkDocs up and running, as you can probably imagine with such a large user base. I desperately hope that the maintainers of MkDocs can find a good solution and sort this problem out in the coming days, weeks, ... months. Sadly, we're already still waiting for 1.6 to be shipped, which includes some bugfixes that I've been waiting for long.
Sigh. I just realized that `page.title` is no longer a string, but rather a dynamic property. Actually, sometimes it is a value (when set in the nav config) and other times it is a dynamic property.  As a dynamic property, it points to a method which calculates and returns a value each time it is referenced. At different points during the build process, it could return a different value. For example, prior to rendering Markdown it would return a different value than after. 

This makes it significantly more work to implement my `Title` class proposal. Rather than addressing this as a thorough new API, perhaps the quick fix (wrap `title_from_render` in a `striptags` call) is the best way forward for now.
@squidfunk you seem to be misunderstanding me. I am not at all suggesting that you (or anyone) should be including `striptag` filters in your templates as a way to get out of fixing this in MkDocs. I agree with you that this issue must be fixed in MkDocs. However, I am also suggesting that you make a change you your templates regardless of how MkDocs fixes this issue. The fact is, page titles could always contain markup, even before 1.5 was released. For example, a user could add markup to a title defined in the nav config, or in a page's metadata. MkDocs does nothing to alter the user provided string in those cases. I suppose in the past any reports would have been handled as user error. However, today plugins can populate page content with a list of page titles (for example) and in that use-case, it is reasonable to preserve markup in the page titles. Therefore, it is reasonable for users to assume that they can include markup in page titles. And, by extension, it is reasonable to expect theme and plugin developers to assume that page titles will have markup in them.

I have just pushed a PR (at #3546) which provides a quick fix (titles coming from rendered Markdown have **all** markup stripped). However, I am still convinced that we need more extensive solution which works for all sources of page titles. That solution should provide an API which makes both "plain text" and "original" versions of a page title available where the "original" version would retain any markup.
Hi.
This thread got needlessly complicated.
It's not necessary to argue *whether* we should do something. Yes we should do something. I have said that I'd just like to receive clear feedback.

Saying that this needs to be fixed and bumping the thread doesn't help, I am aware that there is a problem and I am not moving forward with anything until a good solution is found. Let's also not pretend that in MkDocs 1.4 everything was perfect and `title` could never have HTML in it. Though of course it *can* help to gain insight into what users historically expected from the `title` and keep that in mind for any solution.

Talking about the implementation or a workaround *also* doesn't help. The implementation won't decide the desired solution. And I can code pretty OK anyway.
I did get intrigued by the idea of a dual "HTML-friendly" + "HTML-averse" title but I still think we must first decide on what the main mode of the *existing* `title` should be, regardless of this.

I have said very simply:

> The most helpful feedback that I'd like to see is what value of `title` you expect to see for a given input Markdown.

That's really all. I just want examples of inputs and desired outputs, hopefully with some clarification regarding which rules were applied to reach that output.
So let me re-state my one questionnaire:

1. The document starts with this:

   ```markdown
   # Title *title* <i>title</i> :material-book:
   ```

   *emoji extension is enabled

   What should the value of `title` be? Why?

2. The document starts with this:

   ```markdown
   # \*Hello --- *beautiful* `wor<dl>`
   ```

   *[smarty](https://python-markdown.github.io/extensions/smarty/) extension is enabled

   What should the value of `title` be? Why?

1. `# Title *title* <i>title</i> :material-book:`

    I answered this one previously [here](https://github.com/mkdocs/mkdocs/issues/3357#issuecomment-1722297873) by saying the title should be `Title title title`. The reasoning being that all markup should be stripped. But I've since taken a closer look at the emoji extension and it is less clear. On the one hand, a png or svg doesn't belong in an HTML title (element or attribute). However, the extension can be customized to use Unicode codepoints for those emojis which have a [Unicode counterpart](https://www.unicode.org/emoji/charts/emoji-list.html) (not all do). So, I would modify my answer to state that the Unicode codepoint could be included for any emojis which have one specified in the Markup. Otherwise, it should be excluded altogether. The colon short name (`:material-book:`) should never remain in the output. In other words, `:smile:` could result in `😄` (the Unicode character `U+1F600`), while others would not. Of course, if the extension is configured to not include the Unicode codepoint in any emojis, then they would all be excluded.

    Notice that I state above that the Unicode codepoint "could" be included, not "should." I'm still not convinced it always should be when available. Some fonts don't support them, and while fallback fonts work in CSS, the `title` element gets displayed by the browser's UI (and/or the OS's UI), not as part of the rendered page. In my limited testing, they do work in Chrome on Windows, but the same Unicode character will display using a different style in the page nav than in the browser UI as the exact styling varies by font. Given the many caveats, I will continue to remain okay with them always being stripped.

2. `` # \*Hello --- *beautiful* `wor<dl>` ``

    That is some weird input. There are a lot of different parts covered in that one line. First of all, Markdown, with the `smarty` extension, outputs this:

    ```html
    <h1>*Hello &mdash; <em>beautiful</em> <code>wor&lt;dl&gt;</code></h1>
    ```

    Obviously, the `<h1>`, `<em>`, and `<code>` tags get stripped. The `*` should remain, as it is a plain character. The `&lt;`, `&gt;`, and `&mdash;` should be unescaped back to `<` , `>`, `—` respectively. Although, as it turns out, at least some browsers will convert the HTML entities back to their respective characters in the title (check the interpreted output using the inspect tool and compare to view-source in Chrome for example; I didn't check other browsers). So the output would be this:

    ```html
    *Hello — beautiful wor<dl>
    ```

Except for special casing the emojis, this can all be accomplished with one call to markupsafe's [striptags](https://markupsafe.palletsprojects.com/en/2.1.x/escaping/#markupsafe.Markup.striptags) method:

```python
>>> import markdown, markupsafe
>>> html = markdown.markdown('# \*Hello --- *beautiful* `wor<dl>`', extensions=['smarty'])
>>> markupsafe.Markup(html).striptags()
'*Hello — beautiful wor<dl>'
```

And if the emoji extension is configured to use the [SVG Sprite](https://facelessuser.github.io/pymdown-extensions/extensions/emoji/#default-emoji-generators-svg-sprite) style of output, then it also preserves the Unicode codepoint:

```python
>>> markupsafe.Markup('<svg class="emojione"><description>😄</description><use xlink:href="./../assets/sprites/emojione.sprites.svg#emoji-1f604"></use></svg>').striptags()
😄
```

In all other output options of the extension, the emoji is stripped completely. To me that seems like a reasonable compromise. If a user wants emojis retained as Unicode characters (when available) in titles, then they need to configure the extension to use the SVG Sprite option, otherwise all emojis will be stripped.

And that is what my proposed fix in #3546 does. Well, that is what it does to titles which come from rendered Markdown only. It does not alter any other sources for titles as the changes in 1.5 did not alter that behavior.
> The `&lt;`, `&gt;`, and `&mdash;` should be unescaped back to `<` , `>`, `—` respectively. 

I'm not sure I was right about that. I was focused on the `<title>` element. But the page titles also get used in the nav, and there I would not expect HTML entities to not be unescaped. Perhaps if we had support for dual "HTML-friendly" and "HTML-averse" titles, then unescaping would make sense for the later case. 

> Although, as it turns out, at least some browsers will convert the HTML entities back to their respective characters in the title (check the interpreted output using the inspect tool and compare to view-source in Chrome for example; I didn't check other browsers).

Assuming all modern browsers behave the same here, then preserving HTML entities would be fine. The issue with that is that markupsafe's `striptags` method won't work as it always unescapes HTML entities. Although, the [method](https://github.com/pallets/markupsafe/blob/c7a412f25335dc54f3ace59f8860ff4ee272f5a6/src/markupsafe/__init__.py#L140-L151) is pretty simple to redefine without the unescape call (4 instead of 5 lines of code). Could even import and use markupsafe's regex (although they are marked private in that they start with an underscore). The lib is already a listed dependency after all.
@waylan I'm pretty sure the `<title>` element also expects the value to be escaped. It's not even that modern browsers unescape the title, the title is unescaped at the level of initially reading the HTML everywhere in the same way.

But so under the current way MkDocs works, indeed it seems that unescaping the title string is untenable - it's bad both for the `<title>` and for normal usage within HTML. In this way, the title is only good for use as a normal string for some other processing.

So, with `page.title = "*Hello — beautiful wor<dl>"`
and this theme HTML:
```html
<title>{{ page.title }}</title>
```
right now this blows up.

I was very surprised to learn that MkDocs doesn't enable Jinja autoescaping, which is the way most people use it these days. Currently all interpolated `{{ values }}` need to be escaped explicitly, and *that* is why this blows up. People should've been escaping such values all along everywhere (by writing `{{ page.title |e }}`) if MkDocs doesn't enable autoescaping.

However if we were to enable autoescaping, then this would work perfectly, as the title would be escaped back.

Which sounds like a huuuge breaking change, but it might just work.

* Without autoescaping (current behavior):
  `str('<hr>')` gets interpolated as `<hr>`
  `Markup` makes no difference
* With autoescaping (ideal behavior):
  `Markup('<hr>')` gets interpolated as `<hr>`
  `str('<hr>')` gets interpolated as `&lt;hr&gt;`
  `Markup('&lt;hr&gt;')` gets interpolated as `&lt;hr&gt;`

So I think we have only these 2 options:

* Enable autoescaping in jinja and put this into title: `*Hello — beautiful wor<dl>`
  I implement this in commit https://github.com/oprypin/mkdocs/commit/516b072fb5bccc6618abd9ec8b6e13b7e30bf364 on top of your commit https://github.com/waylan/mkdocs/commit/305a5aa8456fa90dd5712015535584c263eba849
  Notice that I now store the page content as a `Markup` instance - this is like pre-applying the jinja `|safe` filter - to exempt it from escaping because it's actual HTML, unlike everything else.

* Keep the status quo and put this into title: `*Hello &mdash; beautiful wor&lt;dl&gt;`
  Tags will still need to be stripped, so striptags will need to be reimplemented as you're saying. (Or just call `escape` on it again?)
  The string is now ugly and not usable in non-HTML contexts

---

My commit 516b072fb5bccc6618abd9ec8b6e13b7e30bf364 also implements another idea: it extracts text from the `alt` attribute of a tag if available. This will actually make emojis work whether it's configured to use the png or the svg method.

Thanks for the productive discussion @waylan , I got these ideas from your thoughts.
> if we were to enable autoescaping,

I haven't had an opportunity to follow up for a few days, but earlier this week I was thinking the same thing. Autoescaping should be enabled. It is a breaking change and we may need to think about how to make that change graceful, but IMO it should be the default at some point.

One additional option this gives us is to use the `__html__` method in our own custom class (the same API that `marksafe.Markup` uses). As I understand it, when autoescaping is enabled, Jinja will call the `__html__` method of an object if it has one (which skips escaping), but if no such method exists, then it will escape the value. But when autoescaping is off, then the `__html__` method is ignored and the default value is used as-is. Therefore, we could define a class where the instance would contain the non-stripped string and the `__html__` method would return the stripped value. Here is a simplified example:

```python
class Title(str):

    def __html__(self):
        return striptags(self)
```

In this case, `striptags` would need to be our own implementation which did not do any unescaping (actually the `__html__` method could potentially contain the tag stripping code itself, but I'm trying to keep the example simple). It then get's used as follows:

```python
>>> title = Title('*Hello &mdash; <em>beautiful</em> <code>wor&lt;dl&gt;</code>')
>>> str(title)  # <= the behavior when autoescaping is disabled in a template
"*Hello &mdash; <em>beautiful</em> <code>wor&lt;dl&gt;</code>"
>>> markupsafe.escape(title)   # <= the behavior when autoescaping is enabled in a template
"*Hello &mdash; beautiful worl&lt;dl&gt;"
>>> markupsafe.escape(title) == title.__html__()  # <= `escape` just calls the `__html__` method and returns its return value unaltered
True
```

This allows us to keep both the marked up version and have the template render the stripped version all in one object. It does require autoescaping, but it is using the template systems native API. Therefore, if a template author needs the unstripped version of the title, they can do this:

```django
{% autoescape false %}
    title
{% endautoescape %}
    title
```

And that will render as 

```
*Hello &mdash; <em>beautiful</em> <code>wor&lt;dl&gt;</code>

*Hello &mdash; beautiful worl&lt;dl&gt;
```
By the way, even if autoescaping was not enabled, the same `__html__` API works in reverse:

``` django
    title
{% autoescape true %}
    title
{% endautoescape %}
```

The above would render as: 

```
*Hello &mdash; <em>beautiful</em> <code>wor&lt;dl&gt;</code>

*Hello &mdash; beautiful worl&lt;dl&gt;
```

I think that regardless of which way we go with autoescaping, this is the right approach for preserving markup where it is needed (such as [in the nav](https://github.com/Python-Markdown/markdown/issues/1432#issuecomment-1920258459)), but striping it where it is undesirable.
@waylan I'm not fully sure we should go this deep on this, with tangibly different outputs based purely on the context. An easier to understand alternative could also be applied to make the full markup available - just call it `title_html` or whatever. I'll think about this though
I think yes, enabling autoescape is for sure the future that we want to be in. Applying `Markup` as appropriate will finally make it explicit what is HTML and what is text, and plugins will finally be able to properly put text or HTML into attributes as they like.

What helps make this change graceful is that, even if a plugin **does** need to make an edit in response to this, it is always possible to make such an edit without dropping compatibility with older versions of MkDocs. You basically just sprinkle `Markup()` whenever you are producing a string that's intended to be HTML, and that's it.

For themes theoretically there should be 0 changes necessary, but I'll dig more.
There is a directly related ongoing issue to consider - it's this particular use case in mkdocs-material:

https://github.com/squidfunk/mkdocs-material/blob/02bf848fd504b953acb1bdb8aad9531ac052c93a/src/templates/partials/footer.html#L44

so, putting the title into a textual attribute.

```html
<a aria-label="Previous: {{ page.title | e }}"
```

Currently it is very easy to put arbitrary HTML into the title because sadly the titles specified through the `nav` config are actually raw HTML.

```yaml
nav:
  - '<code foo="bar">My cool page</code>': cool_page.md
```

This works and I guess we'll have to keep that behavior because some people do use it.

But coming back to the original issue- the above will be rendered as

```html
<a aria-label="Previous: &lt;code foo=&quot;bar&quot;&gt;My cool page&lt;/code&gt;" ...
```

which is just wrong.
I think regardless, this `|e` should probably be replaced with `|striptags`? And that works out much better both with and without the discussed changes.

---

But let's still step back to the issue with "raw HTML specified in nav yaml config".
It seems like we'll have to do something about it as well, just like we're doing with the titles originating from markdown.
At first I thought it would make sense to use it as-is and put it into a `Markup` object to mark it safe. But it is very much not safe because it breaks the exact use case above in a bad way. It seems that `|e` becomes a no-op with autoescape enabled and the above example gets rendered as this:

```html
<a aria-label="Previous: <code foo="bar">My cool page</code>" ...
```

(**very** bad)

https://github.com/mkdocs/regressions/actions/runs/7747542574/job/21128277531#step:7:132
> I think regardless, this `|e` should probably be replaced with `|striptags`? And that works out much better both with and without the discussed changes.

This is why I suggested using our own custom class which uses the escaping API (defining an `__html__` method). If that class does tag stripping in the `__html__` method, then the template system's escape call results in tag stripping instead. In other words, by implementing my suggestion, the existing template (`{{ page.title | e }}`) would just work with no edits to the template.

That said, these changes would not mean no edits to any templates at all. Turning on autoescaping would affect everything everywhere (of course taking into account Markup instances). But as I explain [here](https://github.com/Python-Markdown/markdown/issues/1432#issuecomment-1924497842) there are still legitimate instances where no escaping or tag stripping should happen and that would require an edit to existing templates to explicitly bypass escaping/stripping by disabling autoescaping for that specific call. For example, my sitemap generating template would need to disable autoescaping for the label to each link, but could leave it enabled everywhere else on that page.

For completeness, I suppose one does not need to use my custom class. You could do this instead:

```
page.html_title = Markup(title)
page.title = Markup(striptags(title))
```

Note that both are wrapped in `Markup` instances as that ensures any escaping in the template is avoided. Then in my sitemap template, I would need to edit the template to reference the `html_title` instead of `title`. 

Neither creates more or less work for the theme devs. But to me, it seems cleaner to just have the one title which has both values and depends on escaping to determine which to return.
@waylan By the way, at the moment it seems we're converging on putting the full HTML into `title`, right?

i.e.
```markdown
# Title *title* <i>title</i> :material-book:
```
```python
page.title = Title("Title <i>title</i> <i>title</i> <svg blob...></svg>")
```


Although actually that kinda kills my idea of extracting text from the `alt` attribute while doing `striptags`.
> at the moment it seems we're converging on putting the full HTML into title, right?

Yes, it would seem so. 

> Although actually that kinda kills my idea of extracting text from the `alt` attribute while doing `striptags`.

So far I have ignored your mentions of alt text. It is not clear to me why you want to extract alt text. Are you suggesting that if there is alt text on a `<h1>` that the alt text should be used rather than the content of the `<h1>` for the plain text title? Yes, for the user to be able to explicitly define both the markup and plain text versions of the title, then we would need two separate objects (or a single custom object which stores them both). However, the code you have in 516b072f simply replaces the title with the alt text and doesn't retain the the content of the `<h1>` at all. I find that unnecessary as the user can define a plain text title in the frontmatter (meta-data) of the page, which takes precedence over the `<h1>` for a page title. We already have enough ways to define a title, lets not add more.
That was about images / emojis.

For this:

```markdown
# My funny story :smiley: - read it
```

This will be the difference between

`My funny story  - read it` and  
 `My funny story 😄 - read it`

https://facelessuser.github.io/pymdown-extensions/extensions/emoji/#default-emoji-generators-png
I'm only following this discussion loosely, but any approach taken to solve this problem must please, please, please not break the entire ecosystem but try to be entirely downward compatible, so the ecosystem can gradually adapt.

So, if a plugin moves to the next version that will include this change, it still works with a theme that's on 1.5, vice versa. We already limited the version range of Material for MkDocs to <1.6, so we can carefully migrate to the next release, but others might not have done so. I got to be honest here: I expect this to break _a lot_ over at Material for MkDocs because our project is probably the biggest downstream project of MkDocs, with dozens of templates and 12 potentially affected plugins. I'm happy to migrate to whatever solution the maintainers of MkDocs have selected to solve the problem reported here, but we and all of our 40k users need time for that, as well as the entire ecosystem. Material for MkDocs is one of the most used projects affiliated with MkDocs out there, and we try very hard to make and keep it easy for our awesome users.

Thank you very much for your understanding and cooperation.
Turns out, the first change to a plugin already breaks the integration with Material for MkDocs. This is exactly what I was referring to in my previous comment. There needs to be a downward compatible migration path. Related:

- https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/pull/126 (this PR is the root cause)
- https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/issues/127
- https://github.com/squidfunk/mkdocs-material/issues/6734
@squidfunk 
I made a mistake in that particular change, sorry about that. Thanks for letting me know.
Yes, I am spending a lot of effort making it downward compatible. Then you come and tell me that I'm so bad for not making it downward compatible when really I made a separate mistake just in that change https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/pull/126
This is not the way to talk, you're just being "holier than thou" over and over, rather than actually saying something helpful.
I think any downstream project maintainer following this thread worries a bit about having to deal with non-easily-fixable (or not fixable-at-all) breaking changes. I also think we know that you (@oprypin and @waylan, since you both seem to be the most invested in this issue) are being careful about backward compatibility. Honestly I'm lost here so thank you for doing this hard work. I think it's safe to assume that downstream maintainers are simply waiting for clear instructions once MkDocs maintainers have reached consensus on a solution, whether there are breaking changes or not.

I'll continue to test PR branches locally in any case and report any issue I find in relation with my own plugins (and I invite every other plugin/theme maintainer to do the same of course) :slightly_smiling_face: Maybe an updated issue body with a call-to-test disclaimer and some communication would bring more downstream maintainers to do just that, once there's a champion PR? Currently it seems there are 4 PRs related to this issue:

- https://github.com/mkdocs/mkdocs/pull/3533
- https://github.com/mkdocs/mkdocs/pull/3546
- https://github.com/mkdocs/mkdocs/pull/3560
- https://github.com/mkdocs/mkdocs/pull/3562

Or maybe it's not needed at all and we should all just keep an eye on the [regressions pipeline](https://github.com/mkdocs/regressions/actions), maybe adding relevant projects to it to make sure each case is covered.

(Feel free to hide this meta comment)
@oprypin

> Then you come and tell me that I'm so bad for not making it downward compatible when really I made a separate mistake just in that change https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/pull/126
This is not the way to talk, you're just being "holier than thou" over and over, rather than actually saying something helpful.

I did not want to hurt you. Being condescending or disrespectful is never my goal. Could you please point me to the exact formulation in my comment that you feel was off? I'm not sure where you read that I thought you or your actions were as you phrase it 'bad'.

Maybe it's good to understand how all of this played out from my vantage point. We started the conversation several months ago, and then nothing happened for a long time. No doubt, this is a complex matter. Waylan came in, and although he had some good points, we did not seem to find an agreement. I retracted myself from the further discussion, knowing that the maintainers of MkDocs will find a good solution for this. I continued to receive email notifications, and the latest changes caught my attention, because they sounded potentially breaking to my work on Material for MkDocs.

Now, as you know, I maintain a project with tens of thousands of users, and as many of the support requests land on our issue tracker, since for many users Material for MkDocs _is_ MkDocs (nothing we promote), I wanted to point out that for the ecosystem to remain successful and growing, IMHO, a clear and easy migration path is essential. Several of the Material for MkDocs major releases where necessary because there were breaking changes in MkDocs in patch- or minor-level versions. The impending changes sound very fundamental to me, as they touch plugins and themes. Since Material for MkDocs is one of the biggest downstream projects, I wanted to raise awareness.

A few hours later, [a user created an issue on our issue tracker](https://github.com/squidfunk/mkdocs-material/issues/6734), pointing out that the git-revision-date-localized plugin integration is broken. A new revision of the plugin got released, which included a commit by you, talking about preparing the plugin for the new approach of escaping in templates. Downgrading the plugin resolved the error. I didn't see any reaction to my comment, so I felt that I needed to raise awareness about the issue, which was exactly the thing I talked about in my previous comment, hoping for some reaction, given that this change affected thousands and thousands of projects, as the git-revision-date-localized plugin has [340k downloads a month](https://pypistats.org/packages/mkdocs-git-revision-date-localized-plugin). In my humble view, fixing this was urgent.

Please tell me which words got you upset, so I can chose them more wisely next time. Additionally, I tried to be helpful, but it seems that my input on this issue, albeit it affects my work deeply, is of no help to you or waylan, as I can read from his and your comments. I'm happy about any suggestion on how you feel we can better gravitate towards a solution.

I hope we can resolve this matter quickly, so we can actually work on the problem ☺️
Here's a summary of what's happening with titles.

---

A title of a **nav** item can come from one of three sources, in this order of precedence.

1. From the `nav` config in mkdocs.yml:
   The string is directly passed to themes, which normally paste it as raw HTML (tags, entities and all)
   This was a bad historical decision, but at least there's no ambiguity. May or may not be worth trying to change this.

2. From the  `meta` `title` in the document:
   Exactly the same as (1)

3. From the first heading in the document (which is **surely** understood to be Markdown)
    * In MkDocs 1.4 and lower:
       The Markdown string is directly passed to themes, which normally paste it as raw HTML (tags, entities and all)
       This is really bad and I don't think anyone expected it.
    * In MkDocs 1.5:
       The Markdown is rendered, all HTML tags removed **but with a bug that causes a failure to remove some of tags**, HTML entities are kept.

---

A title of a **ToC** item comes from any heading in the document:
  * The Markdown is rendered, all HTML tags removed, HTML entities are kept.
     (This is instead done by an implementation from python-markdown, thus with subtle differences)

---

As you can see, MkDocs 1.4 had "consistent" nav item titles in that it consistently put them as raw HTML even though in one of the contexts the source material is clearly Markdown.

Then in MkDocs 1.5 instead we got a different form of consistency: the way to determine a nav item title from a Markdown heading became the same as determining the ToC item title from a Markdown heading (though with subtle differences and a bug).

---

The original bug report correctly identifies this failure to remove some of the tags from ToC titles (highlighted in bold above). But the followup reply then totally wrongly claimed that MkDocs 1.4 was perfect and was stripping tags. But as you can see it really wasn't stripping anything whatsoever, just not bothering to render Markdown or do anything at all otherwise.

---

In terms of the **latest** replies in this thread, they went in this direction:
Modify the nav item determination in the following way:

3. From the first heading in the document (Markdown):
   * The Markdown is rendered and all HTML is kept, although attempts are still made to not break existing usages and somehow magically strip the HTML tags where they weren't expected

And I actually no longer think that this is a feasible way to proceed unless we also have plans to rework how ToC item headings are determined. Also it's otherwise quite reckless. Instead I'm more inclined to just fully unify these two behaviors (remember, the ToC item determination is only subtly different from current MkDocs 1.5 behavior for nav items), meaning that mainly just the bug at the beginning of this thread is to be fixed through better removal of tags. Other changes (such as autoescape for templates) become unnecessary then, and kind of a separate topic, to be considered on its own merits.

The fact that the first two ways to specify the nav item title *do* still produce unguarded raw HTML *is* a big pain, inconsistency and even danger, but maybe we should not be blinded by this. In some ways it is the bigger problem, but in ways of practicality it doesn't show up much and we don't have much choice but to keep existing behavior anyway.
If your proposed fix is implemented, would there still be a possibility of changing the resolution order with which titles are derived? Specifically, I'm thinking about this issue: https://github.com/mkdocs/mkdocs/issues/3532
@ofek Yes that will be https://github.com/mkdocs/mkdocs/issues/3532#issuecomment-1872477066 #3533
It also seems to me that there is a high demand for a good implementation of this functionality:

"Take an etree element in the middle of Markdown rendering, finish rendering it and convert it to a reasonable string with HTML entities but without any HTML tags."

I only now realize that the 'toc' extension [implements exactly this](https://github.com/Python-Markdown/markdown/blob/c334a3e47e6565469344154f966cf0eb9aca0de3/markdown/extensions/toc.py#L314) by necessity and instead I had made two separate implementations of pretty much the same thing:

1. [mkdocs-literate-nav](https://github.com/oprypin/mkdocs-literate-nav/blob/fb047ef72f1a65ca305852933c2a7f100a7279d7/mkdocs_literate_nav/parser.py#L94
) 
    Wrong because it forgets to un-stash stashed HTML, or run other postprocessors
2. [mkdocs 1.5](https://github.com/mkdocs/mkdocs/blob/d6fcc56a3e2b287a2dc0310a7472ba4cdb177854/mkdocs/structure/pages.py#L562)
    Wrong because it forgets to remove tags after unstashing HTML
    
We've been talking how to safely strip tags after unstashing HTML, and that we may need a new implementation, but Python-Markdown [has](https://github.com/Python-Markdown/markdown/blob/c334a3e47e6565469344154f966cf0eb9aca0de3/markdown/extensions/toc.py#L87) that already (though not sure if it's 100% reliable and optimal)

Now, it might just seem like I'm saying that we should just switch every implementation to the one from the 'toc' extension. And that's probably true. But it does have its own deficiencies!

* It runs as a treeprocessor [at priority 5](https://github.com/Python-Markdown/markdown/blob/a63e6f3416f2e4fc7c2d5edb2768ceed9218df20/markdown/extensions/toc.py#L404) and yanks the content before treeprocessors with later priorities had a chance to run. Specifically the 'smarty' treeprocessor ([at priority 2](https://github.com/Python-Markdown/markdown/blob/a63e6f3416f2e4fc7c2d5edb2768ceed9218df20/markdown/extensions/smarty.py#L272)) is what I noticed to be a problem. Currently `foo --- bar` with 'smarty' extension remains just `foo --- bar` in the ToC title. My implementation for the nav titles avoided this bug just by being at a later priority - it got the correct `foo &mdash; bar`.

   I think this is pretty much just a bug to be reported on Python-Markdown.

* I was also saying that ideally `<img alt="foo description" src="foo.png">` would be replaced by "`foo description`" rather than by nothing. And ideally I'd like to see this in both the nav and toc titles.
> The fact that the first two ways to specify the nav item title _do_ still produce unguarded raw HTML _is_ a big pain, inconsistency and even danger, but maybe we should not be blinded by this. In some ways it is the bigger problem, but in ways of practicality it doesn't show up much and we don't have much choice but to keep existing behavior anyway.

You are correct that this hasn't been much of an issue over time. But that is because when a user defines their custom title, they immediately see how it is broken and make a change and remove the offending markup. As the title rarely appears anywhere outside of the navigation and `<title>`, the user doesn't feel the need to add the fancy markup to their titles. But, when the title comes from the body content of a page, the user understandably wants to keep their fancy markup in the body content. And so this is where the issue is prevalent.

That said, there is no reason why we couldn't just as easily sanitize titles regardless of where they come from. In fact, it annoys me that the change in 1.5 removed the setter and getter for `Page.title`. By using a setter and getter, we could easily ensure any page title, regardless of source, is always properly sanitized. For example, when a plugin does `somepage.title = '<strong>custom</strong> title with weird&mdash;markup <svg>...</svg>'`, the setter (or perhaps the getter) could sanitize it.

As I see it, there are really only three issues being discussed here:

1. How to obtain the title from the body content of the page so that it is in a sensible form.
2. How to sanitize the title.
3. Do we want to retain the unsanitized title along with the sanitized title and if so, what API do we implement to make both available?

The first issue does not need to concern itself with the other two. The only concern for the first issue is to ensure a fully rendered string is obtained. One could argue that that is not even on-point for this issue as this issue is really only about issues 2 and 3.

It was issue 3 which sent us off on the autoescape detour. It would be an elegant solution, and perhaps a good long-term goal for MkDocs in general, but that is a big change and this issue needs an immediate fix. For me, the hang-up is what API to use for issue 3 as I personally need the unsanitized title to be retained as I have explained in previous comments.
Meanwhile mkdocs-material wasn't so interested in solving the general challenge of exposing the unsanitized titles and instead has been selling a plugin that replaces the titles with rendered unsanitized versions, for a while already.

https://squidfunk.github.io/mkdocs-material/plugins/typeset/

It might be attractive to publish an equivalent plugin and suggest it to anyone who wants the unsanitized titles. In effect, such a plugin has 0 advantages over just adding a built-in config with this effect. But there's a benefit in terms of existing knowledge and discoverability of such a solution being available as a plugin.
> In fact, it annoys me that the change in 1.5 removed the setter and getter for Page.title

There was never a setter, it was a plain attribute 🤔
> there is no reason why we couldn't just as easily sanitize titles regardless of where they come from

Well for example if the title came from the nav config, then someone had to have intentionally put the markup in there, so unconditionally sanitizing it also doesn't make much sense
> Meanwhile mkdocs-material wasn't so interested in solving the general challenge of exposing the unsanitized titles and instead has been selling a plugin that replaces the titles with rendered unsanitized versions, for a while already.
>
> https://squidfunk.github.io/mkdocs-material/plugins/typeset/
>
> It might be attractive to publish an equivalent plugin and suggest it to anyone who wants the unsanitized titles. In effect, such a plugin has 0 advantages over just adding a built-in config with this effect. But there's a benefit in terms of existing knowledge and discoverability of such a solution being available as a plugin.

We'd be happy to deprecate that plugin, if MkDocs would expose the same functionality. The plugin was the only way we could make it work with reasonable effort. In fact, I started writing plugins because almost all of my feature request over at MkDocs got turned down by the maintainers, but if MkDocs or a plugin maintained by the MkDocs team offers the exact same functionality, we're going to deprecate the [typeset](https://squidfunk.github.io/mkdocs-material/plugins/typeset/) plugin immediately. Always happy to have to maintain less 😉
> > there is no reason why we couldn't just as easily sanitize titles regardless of where they come from
> 
> Well for example if the title came from the nav config, then someone had to have intentionally put the markup in there, so unconditionally sanitizing it also doesn't make much sense

As has been covered multiple times in this discussion, there are use cases for a title with Markup. I'm assuming that the user wants the Markup for those use cases, but at the same time, in some situations (like the `<title>` element), the title should never have any markup. Therefore, I am suggesting that a page accept a title with markup, and then provide an API to retrieve both the title with markup and the title with all markup stripped. I thought we had established that I and was merely pointing out that should happen for regardless of the source of the title (nav config, meta-data, page content, plugin, or other).
FWIW, #3564 did not return to pre-1.5 behavior since it only affected H1; H2-H6 still retain HTML tags in the navigation sidebar, while in 1.4 they did not.
@vedranmiletic You would have to back up that claim, because this does not appear possible to me.
> @vedranmiletic You would have to back up that claim, because this does not appear possible to me.

I apologize, I was wrong and you are indeed right. There are no changes in behavior from pre-1.5 to 1.5 and newer.

To clarify, the issue that I thought of is that leftovers of HTML character references appears in navigation sidebar; for example, see first H2 on [this page](https://group.miletic.net/hr/nastava/materijali/web-arhitekture/). It must be another tool that I remember that didn't have leftovers of character references as I use these Marp presentations in various Markdown tools.

It would be great to have this behavior fixed in some way for H2-H6, but now it doesn't seem to be the same issue as this one. Should I open a separate bug?
This looks like some different theme. You can create a separate issue but make sure to link which theme is used, it's more likely that the theme should be changed
@vedranmiletic Aha yes, I can observe your bug in the default theme with zero extensions.

It is about writing an email in a heading such as `## <foo@example.org>`
-applies to any theme and produces this in ToC: `amp#102;amp#111; ...`

This is a bug in Python-Markdown but a PR is open that happens to fix it:
https://github.com/Python-Markdown/markdown/pull/1441#discussion_r1485682953
@oprypin thanks!

Resolving patch:
diff --git a/mkdocs/structure/pages.py b/mkdocs/structure/pages.py
--- a/mkdocs/structure/pages.py
+++ b/mkdocs/structure/pages.py
@@ -1,6 +1,5 @@
 from __future__ import annotations
 
-import copy
 import enum
 import logging
 import posixpath
@@ -20,6 +19,7 @@
 from mkdocs.structure import StructureItem
 from mkdocs.structure.toc import get_toc
 from mkdocs.utils import _removesuffix, get_build_date, get_markdown_title, meta, weak_property
+from mkdocs.utils.rendering import get_heading_text
 
 if TYPE_CHECKING:
     from xml.etree import ElementTree as etree
@@ -555,23 +555,13 @@ class _ExtractTitleTreeprocessor(markdown.treeprocessors.Treeprocessor):
     def run(self, root: etree.Element) -> etree.Element:
         for el in root:
             if el.tag == 'h1':
-                # Drop anchorlink from the element, if present.
-                if len(el) > 0 and el[-1].tag == 'a' and not (el[-1].tail or '').strip():
-                    el = copy.copy(el)
-                    del el[-1]
-                # Extract the text only, recursively.
-                title = ''.join(el.itertext())
-                # Unescape per Markdown implementation details.
-                title = markdown.extensions.toc.stashedHTML2text(
-                    title, self.md, strip_entities=False
-                )
-                self.title = title.strip()
+                self.title = get_heading_text(el, self.md)
             break
         return root
 
     def _register(self, md: markdown.Markdown) -> None:
         self.md = md
-        md.treeprocessors.register(self, "mkdocs_extract_title", priority=-1)  # After the end.
+        md.treeprocessors.register(self, "mkdocs_extract_title", priority=1)  # Close to the end.
 
 
 class _AbsoluteLinksValidationValue(enum.IntEnum):
diff --git a/mkdocs/utils/rendering.py b/mkdocs/utils/rendering.py
new file mode 100644
--- /dev/null
+++ b/mkdocs/utils/rendering.py
@@ -0,0 +1,104 @@
+from __future__ import annotations
+
+import copy
+from typing import TYPE_CHECKING, Callable
+
+import markdown
+import markdown.treeprocessors
+
+if TYPE_CHECKING:
+    from xml.etree import ElementTree as etree
+
+# TODO: This will become unnecessary after min-versions have Markdown >=3.4
+_unescape: Callable[[str], str]
+try:
+    _unescape = markdown.treeprocessors.UnescapeTreeprocessor().unescape
+except AttributeError:
+    _unescape = lambda s: s
+
+# TODO: Most of this file will become unnecessary after https://github.com/Python-Markdown/markdown/pull/1441
+
+
+def get_heading_text(el: etree.Element, md: markdown.Markdown) -> str:
+    el = copy.deepcopy(el)
+    _remove_anchorlink(el)
+    _remove_fnrefs(el)
+    _extract_alt_texts(el)
+    return _strip_tags(_render_inner_html(el, md))
+
+
+def _strip_tags(text: str) -> str:
+    """Strip HTML tags and return plain text. Note: HTML entities are unaffected."""
+    # A comment could contain a tag, so strip comments first
+    while (start := text.find('<!--')) != -1 and (end := text.find('-->', start)) != -1:
+        text = text[:start] + text[end + 3 :]
+
+    while (start := text.find('<')) != -1 and (end := text.find('>', start)) != -1:
+        text = text[:start] + text[end + 1 :]
+
+    # Collapse whitespace
+    text = ' '.join(text.split())
+    return text
+
+
+def _render_inner_html(el: etree.Element, md: markdown.Markdown) -> str:
+    # The `UnescapeTreeprocessor` runs after `toc` extension so run here.
+    text = md.serializer(el)
+    text = _unescape(text)
+
+    # Strip parent tag
+    start = text.index('>') + 1
+    end = text.rindex('<')
+    text = text[start:end].strip()
+
+    for pp in md.postprocessors:
+        text = pp.run(text)
+    return text
+
+
+def _remove_anchorlink(el: etree.Element) -> None:
+    """Drop anchorlink from the element, if present."""
+    if len(el) > 0 and el[-1].tag == 'a' and el[-1].get('class') == 'headerlink':
+        del el[-1]
+
+
+def _remove_fnrefs(root: etree.Element) -> None:
+    """Remove footnote references from the element, if any are present."""
+    for parent in root.findall('.//sup[@id]/..'):
+        _replace_elements_with_text(parent, _predicate_for_fnrefs)
+
+
+def _predicate_for_fnrefs(el: etree.Element) -> str | None:
+    if el.tag == 'sup' and el.get('id', '').startswith('fnref'):
+        return ''
+    return None
+
+
+def _extract_alt_texts(root: etree.Element) -> None:
+    """For images that have an `alt` attribute, replace them with this content."""
+    for parent in root.findall('.//img[@alt]/..'):
+        _replace_elements_with_text(parent, _predicate_for_alt_texts)
+
+
+def _predicate_for_alt_texts(el: etree.Element) -> str | None:
+    if el.tag == 'img' and (alt := el.get('alt')):
+        return alt
+    return None
+
+
+def _replace_elements_with_text(
+    parent: etree.Element, predicate: Callable[[etree.Element], str | None]
+) -> None:
+    """For each child element, if matched, replace it with the text returned from the predicate."""
+    carry_text = ""
+    for child in reversed(parent):  # Reversed for the ability to mutate during iteration.
+        # Remove matching elements but carry any `tail` text to preceding elements.
+        new_text = predicate(child)
+        if new_text is not None:
+            carry_text = new_text + (child.tail or "") + carry_text
+            parent.remove(child)
+        elif carry_text:
+            child.tail = (child.tail or "") + carry_text
+            carry_text = ""
+    if carry_text:
+        parent.text = (parent.text or "") + carry_text


Test patch:
diff --git a/mkdocs/tests/structure/page_tests.py b/mkdocs/tests/structure/page_tests.py
--- a/mkdocs/tests/structure/page_tests.py
+++ b/mkdocs/tests/structure/page_tests.py
@@ -342,6 +342,12 @@ def test_page_title_from_setext_markdown(self):
             expected='Welcome to MkDocs Setext',
         )
 
+    def test_page_title_from_markdown_with_email(self):
+        self._test_extract_title(
+            '''# <foo@example.org>''',
+            expected='&#102;&#111;&#111;&#64;&#101;&#120;&#97;&#109;&#112;&#108;&#101;&#46;&#111;&#114;&#103;',
+        )
+
     def test_page_title_from_markdown_stripped_anchorlinks(self):
         self._test_extract_title(
             self._SETEXT_CONTENT,
@@ -349,6 +355,24 @@ def test_page_title_from_markdown_stripped_anchorlinks(self):
             expected='Welcome to MkDocs Setext',
         )
 
+    def test_page_title_from_markdown_strip_footnoteref(self):
+        foootnotes = '''\n\n[^1]: foo\n[^2]: bar'''
+        self._test_extract_title(
+            '''# Header[^1] foo[^2] bar''' + foootnotes,
+            extensions={'footnotes': {}},
+            expected='Header foo bar',
+        )
+        self._test_extract_title(
+            '''# *Header[^1]* *foo*[^2]''' + foootnotes,
+            extensions={'footnotes': {}},
+            expected='Header foo',
+        )
+        self._test_extract_title(
+            '''# *Header[^1][^2]s''' + foootnotes,
+            extensions={'footnotes': {}},
+            expected='*Headers',
+        )
+
     def test_page_title_from_markdown_strip_formatting(self):
         self._test_extract_title(
             '''# \\*Hello --- *beautiful* `wor<dl>`''',
@@ -356,17 +380,19 @@ def test_page_title_from_markdown_strip_formatting(self):
             expected='*Hello &mdash; beautiful wor&lt;dl&gt;',
         )
 
+    def test_page_title_from_markdown_html_entity(self):
+        self._test_extract_title('''# Foo &lt; &amp; bar''', expected='Foo &lt; &amp; bar')
+        self._test_extract_title('''# Foo > & bar''', expected='Foo &gt; &amp; bar')
+
     def test_page_title_from_markdown_strip_raw_html(self):
-        self._test_extract_title(
-            '''# Hello <b>world</b>''',
-            expected='Hello world',
-        )
+        self._test_extract_title('''# Hello <b>world</b>''', expected='Hello world')
+
+    def test_page_title_from_markdown_strip_comments(self):
+        self._test_extract_title('''# foo <!-- comment with <em> --> bar''', expected='foo bar')
 
     def test_page_title_from_markdown_strip_image(self):
-        self._test_extract_title(
-            '''# Hi ![😄](hah.png)''',
-            expected='Hi',  # TODO: Should the alt text of the image be extracted?
-        )
+        self._test_extract_title('''# Hi ![😄](hah.png)''', expected='Hi 😄')
+        self._test_extract_title('''# Hi *-![😄](hah.png)-*''', expected='Hi -😄-')
 
     _ATTRLIST_CONTENT = dedent(
         '''