Repository: lepture/mistune

Issue:
Certain characters in inline code incorrectly parsed (e.g., `&`)
MWE:

```python
import mistune
from mistune.core import BlockState
markdown = mistune.create_markdown(renderer="ast")

md = r"`&<>`"
tokens = markdown(md)

print(tokens)
```

Output:

```python
[{'type': 'paragraph', 'children': [{'type': 'codespan', 'raw': '&amp;&lt;&gt;'}]}]
```

Hints from issue comments:
@torokati44: We also encountered this. The cause is 8452faf345a152a149e7f79243fd9693a06ed0e9, more specifically [this change](https://github.com/lepture/mistune/commit/8452faf345a152a149e7f79243fd9693a06ed0e9#diff-f50bd4f1c9941471f448b19c10e0cc1467033977494b1013ae5aa6e9f863cf08L381-R382), I think.
Putting HTML escaping into the _parser_ stage, independently of the output format, is incorrect.@veenstrajelmer: I also encountered this, originally found in https://github.com/omnilib/sphinx-mdinclude/issues/19. I did some digging and found that the issue lays in `mistune.util.escape_url` indeed as @torokati44 points towards, but only because its behavior is different depending on whether `html` is installed (this happens in the import, **at least of mistune 2.0.5**):
https://github.com/lepture/mistune/blob/cb580e89e67ef9b4827daf43afd2058bdf5e58d2/mistune/util.py#L22-L31

Some code to reproduce:
```python
from urllib.parse import quote
import html

link = "https://sonarcloud.io/api/project_badges/measure?project=Deltares_ddlpy&metric=alert_status"

# code from: def escape_url(link):
safe = (
    ':/?#@'           # gen-delims - '[]' (rfc3986)
    '!$&()*+,;='      # sub-delims - "'" (rfc3986)
    '%'               # leave already-encoded octets alone
)
out_nonhtml = quote(link.encode('utf-8'), safe=safe)
out_withhtml = html.escape(quote(html.unescape(link), safe=safe))
out_withhtml_noescape = quote(html.unescape(link), safe=safe)

print(out_nonhtml)
print(out_withhtml)
print(out_withhtml_noescape)
```

This gives different results. The first one is returned if `html` is not installed, the second one if `html` is installed. The third one is correct again:
```
https://sonarcloud.io/api/project_badges/measure?project=Deltares_ddlpy&metric=alert_status
https://sonarcloud.io/api/project_badges/measure?project=Deltares_ddlpy&amp;metric=alert_status
https://sonarcloud.io/api/project_badges/measure?project=Deltares_ddlpy&metric=alert_status
```

Therefore, what I think should fix the issue is to remove the `html.escape()` from the `escape_url` function so the behaviour is always consistent and according to expectations.

**Update for newer mistune versions**
The `escape_url` function looks different in the master branch (>3.0.0):
https://github.com/lepture/mistune/blob/93fd197c6c24353011974110378b738596cde5a1/src/mistune/util.py#L32-L39
In this case omitting `escape` does the trick.
@veenstrajelmer: Also today, in the master branch, `&` still gets converted to `&amp;`. I am not sure whether `escape` should just be removed from `escape_url`, since that does not seem to make sense given the function name. However, this was the behavior before if `html` was not installed with `return quote(link.encode('utf-8'), safe=safe)`, so it might also just be the fix. Either way, it would be great if `&` markdown urls is not converted.@itcarroll: I'm going down the rabbit hole from jupyter-book > sphinx > myst > ??? here, maybe? I'm getting `&amp;` in markdown links that include query parameters. Any workaround?@veenstrajelmer: I think what you are encountering is because of the bug described in this issue indeed. No workaround as far as I know, @lepture could you ahare your thoughts on this discussion?@mentalisttraceur: @itcarroll actually, Jupyter's MyST uses [`markdown-it-py`](https://pypi.org/project/markdown-it-py), not Mistune.@mentalisttraceur: @veenstrajelmer I think you've spotted a *separate* issue. Similar, but independent code paths. The premature HTML-escaping of inline *code* just uses `escape`, totally bypassing `escape_url`.@veenstrajelmer: @mentalisttraceur ok fair, but will this issue be resolved? I see limited response from maintainers of this repository, so I am not sure what to expect. If it is a different issue, does it help if I create a different issue, or will it not matter too much? I am a bit hesitant in putting even more investigation time in this package, because of te limited response. My issue is just a single readme badge that does not work in the docs of all my packages (e.g. https://deltares.github.io/dfm_tools). I can relatively easily convert all the readme's from markdown to rst, but only raised this issue since I prefer markdown.@stdedos: @veenstrajelmer "not having this discussion here", would matter _at least_ for us that are "actively" participating on this issue 🤷
You can also clearly see that someone decided to step up and fix this issue (after this issue being stale for a while).

I would be optimistic - as long as issues don't get "more tangled".@veenstrajelmer: I was not intending to tangle issues, just noticed this issue that seems pretty much as what I am running into. I do not see how they are caused and therefore also not see how they are unrelated. To me they both come down to incorrect character escaping in mistune, but apparently they have different causes. Either way, I have created another issue here: https://github.com/lepture/mistune/issues/394.

Solution patch:
diff --git a/src/mistune/core.py b/src/mistune/core.py
index 07d79bd..a0b2c33 100644
--- a/src/mistune/core.py
+++ b/src/mistune/core.py
@@ -1,4 +1,5 @@
 import re
+import sys
 from typing import (
     Any,
     Callable,
@@ -16,9 +17,9 @@ from typing import (
     Union,
     cast,
 )
-try:
+if sys.version_info >= (3, 11):
     from typing import Self
-except ImportError:
+else:
     from typing_extensions import Self
 
 _LINE_END = re.compile(r'\n|$')
diff --git a/src/mistune/inline_parser.py b/src/mistune/inline_parser.py
index 21b04c1..97423db 100644
--- a/src/mistune/inline_parser.py
+++ b/src/mistune/inline_parser.py
@@ -19,7 +19,7 @@ from .helpers import (
     parse_link_text,
     unescape_char,
 )
-from .util import escape, escape_url, unikey
+from .util import escape_url, unikey
 
 PAREN_END_RE = re.compile(r'\s*\)')
 
@@ -310,7 +310,7 @@ class InlineParser(Parser[InlineState]):
             if len(code.strip()):
                 if code.startswith(' ') and code.endswith(' '):
                     code = code[1:-1]
-            state.append_token({'type': 'codespan', 'raw': escape(code)})
+            state.append_token({'type': 'codespan', 'raw': code})
             return end_pos
         else:
             state.append_token({'type': 'text', 'raw': marker})
diff --git a/src/mistune/renderers/html.py b/src/mistune/renderers/html.py
index 0d999a7..a8d2c47 100644
--- a/src/mistune/renderers/html.py
+++ b/src/mistune/renderers/html.py
@@ -91,7 +91,7 @@ class HTMLRenderer(BaseRenderer):
         return s + ' />'
 
     def codespan(self, text: str) -> str:
-        return '<code>' + text + '</code>'
+        return '<code>' + escape_text(text) + '</code>'
 
     def linebreak(self) -> str:
         return '<br />\n'


Test patch:
diff --git a/tests/test_misc.py b/tests/test_misc.py
index f57c53b..5e21a1a 100644
--- a/tests/test_misc.py
+++ b/tests/test_misc.py
@@ -77,7 +77,7 @@ class TestMiscCases(TestCase):
 
     def test_ast_output(self):
         md = mistune.create_markdown(escape=False, renderer=None)
-        text = '# h1\n\nfoo **bar**'
+        text = '# h1\n\nfoo **bar**\n\n`&<>"`'
         result = md(text)
         expected = [
             {
@@ -94,6 +94,13 @@ class TestMiscCases(TestCase):
                     {'type': 'strong', 'children': [{'type': 'text', 'raw': 'bar'}]}
                 ]
             },
+            {'type': 'blank_line'},
+            {
+                'type': 'paragraph',
+                'children': [
+                    {'type': 'codespan', 'raw': '&<>"'},
+                ]
+            },
         ]
         self.assertEqual(result, expected)