Repository: facelessuser/pymdown-extensions

Issue:
Compatibility issues with Highlighting Lines in Code blocks
### Description

I have a paragraph in my markdown text like this, and when I run it, it parses incorrectly, causing the rendered page to be completely misaligned, here's the source file: [TestOne.md](https://github.com/user-attachments/files/18033885/TestOne.md).

````
```javascript [4]
function hello(name) {
    let phrase = `Hello, ${name}!`;

    debugger;

    say(phrase);
}
```
````

<mark>It would be rendered like this:</mark>

<img width="719" alt="iShot_2024-12-06_14 32 12" src="https://github.com/user-attachments/assets/19f04566-4dd8-4eef-9d57-4e66ab1552d7">

---

<mark>This is the correct parsing in Typora：</mark>

<img width="986" alt="iShot_2024-12-06_14 32 36" src="https://github.com/user-attachments/assets/e8bf790c-3f25-47d1-87aa-5534864372f8">

<br />The reason is that the [4] after the token is not recognized, and it treats the [4] as the beginning of the code, which should normally be the next line.

I know this is not the canonical writeup for this project, but it could be designed so that ignore unrecognized markup when it is encountered, thus not affecting the parsing and rendering of later text.This writeup renders fine in other editors, such as Typora, Github and so on.

This writeup is from other project [revealjs](https://revealjs.com/code/), I compared, this design is much simpler, just need to set the number, don't need to remember the property name, and it also supports step-by-step highlights.

- If you agree with this design, you can consider adding support in subsequent updates;
- If other settings are involved and it is troublesome to modify, then forget it, but it needs to be designed to ignore the markup and not affect the analysis and rendering of the following text;

Here are some write-ups for comparison and reference:

<img width="966" alt="iShot_2024-12-05_12 40 08" src="https://github.com/user-attachments/assets/1aba0412-8041-4e47-b4eb-efb6cf3958a0">

### Minimal Reproduction

unnecessary

### Version(s) & System Info

unnecessary

Discussion:
@facelessuser: I'll start off by saying that while this was filed as a bug, it is not a bug and the docs clearly state the parsing expectations for this [extension](https://facelessuser.github.io/pymdown-extensions/extensions/superfences/). I would consider this a feature request or an enhancement at the very least.

Additionally, I will state that this extension is not meant to support every variation of code blocks created in the past, and certainly is not designed with the requirement to adopt all new syntaxes that various parsers may decide upon in the future.

SuperFences was based on the syntax of [CodeHilite](https://python-markdown.github.io/extensions/code_hilite/) which has been around probably longer than all the listed examples' implementations. A lot of the examples are probably based on CommonMark which came out much later. Python Markdown is an old-school parser and is not expected to conform to any of the expectations of CommonMark. SuperFences specifically based the syntax on CodeHilite to provide compatibility when people came over from the Python Markdown default provided extension.

It should also be noted that SuperFences provides a generic way to create custom fences to parse unaltered blocks of text. This was mainly done because Python Markdown's implementation of code handling is quite a bit cumbersome due to its need to be implemented as a preprocessor. Providing a generic interface that people could tap into meant others could leverage the work and that I'd never have to rewrite that logic again if I needed to pull off similar logic.

While primarily used for code blocks, it should be noted that the logic is reused for non-code block custom fences. Adding a line highlight specific syntax like `[num]` would only be used by code highlighting and would be redundant functionality.

Now, regarding the aforementioned example:

````md
```javascript [4]
function hello(name) {
    let phrase = `Hello, ${name}!`;

    debugger;

    say(phrase);
}
```
````

I think SuperFences logic here is expected and logical. In order to have a fenced code block, the header must be well-formed and conform to expectations. If expectations are not met, the block will be treated as a paragraph, and then the inline code highlighter will be allowed to parse sections within the paragraph, which is exactly what happens. I don't find the output surprising at all. The difference is that GitHub (and possibly some other parsers) will look at the header and, as long as the header doesn't include another `` ` ``, it will accept all the other junk on the line. This is certainly a choice, but one that I do not agree with.

Is there merit in the new numerical syntax? I will generally say yes. There is certainly nothing wrong with the syntax `[num]`, and I can certainly see it being a subjectively preferred approach, but we do already have a convention, and our aim was compatibility with CodeHilite, and being generic enough so that people could create their own custom fences.

As far as accepting anything in a code fence header, I do not like the idea of code blocks swallowing any and all junk on the fenced code block header line. Having a clear indication when you are providing nonsense to the parser is something that I like, and the parser failing to parse the content as a fence code block is that indicator.

I admit that this is an opinionated viewpoint, but these are, to some extent, opinionated extensions.
@jaywhj: @facelessuser 

Okay, I got it. Thank you for your response, very detailed.

My intention is not to ask you to change a new parsing method for this project, just to ask if it's possible to ignore the parsing of this part of the middle bracket [] so that it doesn't affect the later rendering.

I think it should be easy to handle, for example, \`\`\`java  [3], [3] is preceded by some space, you can split the line with a space, and then just take the first part as a syntax markup, and ignore anything after that line. Github, Typora, and other parsers seem to work this way. @facelessuser: > My intention is not to ask you to change a new parsing method for this project, just to ask if it's possible to ignore the parsing of this part of the middle bracket [] so that it doesn't affect the later rendering.

I wasn't sure. It seemed to mention it as a possible approach with a fallback to ignoring the content if implementing was not desired. Obviously, this was my misinterpretation 🙂.

> I think it should be easy to handle, for example, ```java [3], [3] is preceded by some space, you can split the line with a space, and then just take the first part as a syntax markup, and ignore anything after that line. Github, Typora, and other parsers seem to work this way.

No, I understand this part, but as I detailed earlier, I'm not sure I agree with this approach. These implementations make a number of assumptions. If the line starts with ` ``` ` (or more)  and has no other `` ` `` on the line, then it blindly accepts the entire line as the start of a fenced code block. But this might not be the case as inline code can span multiple lines within a paragraph.

Additionally, as stated earlier, I'm not sure I'm okay with hiding bad syntax on the header line and confusing people by making them think their header is valid, but then the functionality they expected isn't shown in the output. Part of my reasoning is preventing people from filing issues here. If they see the code block not rendered properly, they may take a closer look at their syntax. If this prevents even a handful of filed issues here that I have to respond to, it is worth it.

@jaywhj: Okay, thank you very much.
On the premise of not affecting the existing functions, it would be nice if it could be compatible with this writing, after all, other mainstream editors  can parse and render normally 😁.@facelessuser: > after all, other mainstream editors can parse and render normally

Yes, specifically a number of CommonMark parsers mainly, but many that are not do not.

> it would be nice if it could be compatible with this writing

Sure. I'll say this, if it is something I can make "opt-in", without too much complexity, then I might consider it. I'll leave this as a "maybe"  with low priority for now.

Solution patch:
diff --git a/docs/src/markdown/about/changelog.md b/docs/src/markdown/about/changelog.md
index 594e75a48..29fd2ef34 100644
--- a/docs/src/markdown/about/changelog.md
+++ b/docs/src/markdown/about/changelog.md
@@ -1,7 +1,10 @@
 # Changelog
 
-## Unreleased
+## 10.15.0
 
+-   **NEW**: SuperFences: Add `relaxed_headers` option which can tolerate bad content in the fenced code header. When
+    enabled, code blocks with bad content in the header will likely still convert into code blocks, often respecting
+    the specified language.
 -   **FIX**: Blocks: Fix some corner cases of nested blocks with lists.
 -   **FIX**: Tab and Tabbed: Fix a case where tabs could fail if `combine_header_slug` was enabled and there was no
     header.
diff --git a/docs/src/markdown/extensions/superfences.md b/docs/src/markdown/extensions/superfences.md
index b92d5b4ab..85293f4c8 100644
--- a/docs/src/markdown/extensions/superfences.md
+++ b/docs/src/markdown/extensions/superfences.md
@@ -611,6 +611,54 @@ import foo
 ```
 ///
 
+## Relaxed Headers
+
+/// new | New in 10.15
+///
+
+The `relaxed_headers` [option][#options] is a feature that can be enabled which will cause SuperFences to treat fenced
+code headers in a more relaxed manner.
+
+The default approach of SuperFences is to bail on processing a fenced code if the there is invalid content in a fenced
+header. This may be because you are processing fenced code blocks from other, incompatible Markdown parsers. Having a
+way to treat incompatible code blocks as code blocks, even if they have invalid syntax may be desirable.
+
+As an example, some Markdown parsers may allow for specifying line highlights with `[number]`. This is not supported in
+SuperFences, but if we were processing such a block, with `relaxed_headers` enabled, we would treat it as an actual
+code block ignoring the invalid syntax instead of assuming the entire code block was invalid.
+
+````text title="Bad Code Header Syntax"
+```python [3]
+import code
+
+code.do_stuff()
+```
+````
+
+/// html | div.result
+````md-render
+---
+extensions:
+- pymdownx.highlight
+- pymdownx.superfences
+
+extension_configs:
+  pymdownx.superfences:
+    relaxed_headers: true
+---
+```python [3]
+import code
+
+code.do_stuff()
+```
+````
+///
+
+It should be noted that this mode does not always guarantee that your code language will be picked up or that relevant
+classes, IDs, attributes, or options will be recognized if invalid content is present, only that it will try and process
+the block as a code block. Additionally, a custom block that explicitly throws a `SuperFencesException` will still have
+that exception raised which will terminate Markdown processing.
+
 ## Custom Fences
 
 SuperFences allows defining custom fences for special purposes. For instance, we could create special fences for
@@ -911,3 +959,4 @@ Option                         | Type         | Default       | Description
 `disable_indented_code_blocks` | bool         | `#!py3 False` | Disables Python Markdown's indented code block parsing.  This is nice if you only ever use fenced blocks.
 `custom_fences`                | [dictionary] | `#!py3 []`    | Custom fences.
 `preserve_tabs`                | bool         | `#!py3 False` | Experimental feature that preserves tabs in fenced code blocks.
+`relaxed_headers`              | bool         | `#!py3 False` | Enable relaxed fenced code headers that are more forgiving of invalid content in the fenced header.
diff --git a/docs/src/mkdocs.yml b/docs/src/mkdocs.yml
index 5f81482fa..63176e9c9 100644
--- a/docs/src/mkdocs.yml
+++ b/docs/src/mkdocs.yml
@@ -108,6 +108,7 @@ markdown_extensions:
   - markdown.extensions.md_in_html:
   - pymdownx.superfences:
       preserve_tabs: true
+      relaxed_headers: true
       custom_fences:
         # Mermaid diagrams
         - name: diagram
diff --git a/mkdocs.yml b/mkdocs.yml
index 2c3711288..be1e844fb 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -108,6 +108,7 @@ markdown_extensions:
   - markdown.extensions.md_in_html:
   - pymdownx.superfences:
       preserve_tabs: true
+      relaxed_headers: true
       custom_fences:
         # Mermaid diagrams
         - name: diagram
diff --git a/pymdownx/superfences.py b/pymdownx/superfences.py
index 0248c8522..0b5656410 100644
--- a/pymdownx/superfences.py
+++ b/pymdownx/superfences.py
@@ -27,7 +27,6 @@
 
 License: [BSD](http://www.opensource.org/licenses/bsd-license.php)
 """
-
 from markdown.extensions import Extension
 from markdown.preprocessors import Preprocessor
 from markdown.blockprocessors import CodeBlockProcessor
@@ -51,6 +50,9 @@
             (?:
                 (?:[ \t]*[a-zA-Z][a-zA-Z0-9_]*(?:=(?P<quot>"|').*?(?P=quot))?)(?=[\t ]|$)  # Options
             )+
+        ) |
+        (?P<unrecognized>
+            (?:([ \t]*[^\s]+)(?=[\t ]|$))+
         )
     )?[ \t]*$
     '''
@@ -231,7 +233,8 @@ def __init__(self, *args, **kwargs):
                 "if nothing is set. - "
                 "Default: ''"
             ],
-            'preserve_tabs': [False, "Preserve tabs in fences - Default: False"]
+            'preserve_tabs': [False, "Preserve tabs in fences - Default: False"],
+            'relaxed_headers': [False, "Relaxed fenced code headers - Default: False"]
         }
         super().__init__(*args, **kwargs)
 
@@ -376,6 +379,7 @@ def get_hl_settings(self):
             css_class = self.config['css_class']
             self.css_class = css_class if css_class else config['css_class']
 
+            self.relaxed_headers = self.config.get('relaxed_headers', False)
             self.extend_pygments_lang = config.get('extend_pygments_lang', None)
             self.guess_lang = config['guess_lang']
             self.pygments_style = config['pygments_style']
@@ -594,9 +598,9 @@ def parse_options(self, m):
         self.formatter = None
         values = {}
         if string:
-            for m in RE_OPTIONS.finditer(string):
-                key = m.group('key')
-                value = m.group('value')
+            for m2 in RE_OPTIONS.finditer(string):
+                key = m2.group('key')
+                value = m2.group('value')
                 if value is None:
                     value = key
                 values[key] = value
@@ -620,8 +624,48 @@ def parse_options(self, m):
                     self.options = options
                     break
 
+        if not okay and self.relaxed_headers:
+            return self.handle_unrecognized(m)
+
         return okay
 
+    def handle_unrecognized(self, m):
+        """Handle unrecognized code headers."""
+
+        okay = False
+        if not self.relaxed_headers:
+            return okay
+
+        if m.group('lang'):
+            self.lang = m.group('lang')
+
+        self.options = {}
+        self.attrs = {}
+        self.formatter = None
+
+        # Run per language validator
+        for entry in reversed(self.extension.superfences):
+            if entry["test"](self.lang):
+                options = {}
+                attrs = {}
+                validator = entry.get("validator", functools.partial(_validator, validator=default_validator))
+                try:
+                    okay = validator(self.lang, {}, options, attrs, self.md)
+                except SuperFencesException:
+                    raise
+                except Exception:
+                    pass
+                if okay:
+                    self.formatter = entry.get("formatter")
+                    self.options = options
+                    if self.attr_list:
+                        self.attrs = attrs
+                    break
+
+        if not okay:
+            self.lang = None  # pragma: no cover
+        return True
+
     def handle_attrs(self, m):
         """Handle attribute list."""
 
@@ -664,6 +708,9 @@ def handle_attrs(self, m):
                         self.attrs = attrs
                     break
 
+        if not okay and self.relaxed_headers:
+            return self.handle_unrecognized(m)  # pragma: no cover
+
         return okay
 
     def search_nested(self, lines):
@@ -683,7 +730,9 @@ def search_nested(self, lines):
                 if m is not None:
 
                     # Parse options
-                    if m.group('attrs'):
+                    if m.group('unrecognized'):
+                        okay = self.handle_unrecognized(m)
+                    elif m.group('attrs'):
                         okay = self.handle_attrs(m)
                     else:
                         okay = self.parse_options(m)


Test patch:
diff --git a/tests/test_extensions/test_superfences.py b/tests/test_extensions/test_superfences.py
index 16eff4ce0..db4aa9991 100644
--- a/tests/test_extensions/test_superfences.py
+++ b/tests/test_extensions/test_superfences.py
@@ -1634,3 +1634,154 @@ def test_broken_brace(self):
             ''',
             True
         )
+
+
+class TestHighlightRelaxedHeaders(util.MdCase):
+    """Test relaxed header cases."""
+
+    extension = ['pymdownx.highlight', 'pymdownx.superfences', 'attr_list']
+    extension_configs = {
+        'pymdownx.superfences': {
+            'relaxed_headers': True,
+            'custom_fences': [
+                {
+                    'name': 'test',
+                    'class': 'test',
+                    'format': custom_format,
+                    'validator': custom_validator_except
+                },
+                {
+                    'name': 'test2',
+                    'class': 'test',
+                    'format': custom_format,
+                    'validator': custom_validator_exploder
+                }
+            ]
+        }
+    }
+
+    def test_custom_fail_exception(self):
+        """Test custom fences forced exception."""
+
+        with self.assertRaises(SuperFencesException):
+            self.check_markdown(
+                r'''
+                ```test [1]
+                test
+                ```
+                ''',
+                '',
+                True
+            )
+
+    def test_custom_fail_exception_relaxed(self):
+        """Test custom fences relaxed forced exception."""
+
+        self.check_markdown(
+            r'''
+            ```test2 [1]
+            test
+            ```
+            ''',
+            '''
+            <div class="highlight"><pre><span></span><code>test
+            </code></pre></div>
+            ''',
+            True
+        )
+
+    def test_bad_lang(self):
+        """Test bad language."""
+
+        self.check_markdown(
+            r'''
+            ```bad
+            test
+            ```
+            ''',
+            '''
+            <div class="highlight"><pre><span></span><code>test
+            </code></pre></div>
+            ''',
+            True
+        )
+
+    def test_bad_option(self):
+        """Test bad options."""
+
+        self.check_markdown(
+            r'''
+            ```pycon bad="My title"
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_bad_attribute(self):
+        """Test bad attribute."""
+
+        self.check_markdown(
+            r'''
+            ```pycon {.class <nonsense>!!!}
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="class highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_bad_attribute_full(self):
+        """Test bad attribute full."""
+
+        self.check_markdown(
+            r'''
+            ```{.pycon .class <nonsense>!!!}
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="class highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_bad_header(self):
+        """Test bad header."""
+
+        self.check_markdown(
+            r'''
+            ```pycon [nonsense]
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="highlight"><pre><span></span><code><span class="gp">&gt;&gt;&gt; </span><span class="kn">import</span><span class="w"> </span><span class="nn">test</span>
+            </code></pre></div>
+            ''',  # noqa: E501
+            True
+        )
+
+    def test_nonsense(self):
+        """Test only nonsense."""
+
+        self.check_markdown(
+            r'''
+            ```[nonsense]
+            >>> import test
+            ```
+            ''',
+            r'''
+            <div class="highlight"><pre><span></span><code>&gt;&gt;&gt; import test
+            </code></pre></div>
+            ''',
+            True
+        )