Repository: facelessuser/pymdown-extensions

Issue:
Snippet IDs with dashes and underscores
### Description

I tried to use snippet IDs with dashes and / or underscores.
Unfortunately it lead to the whole snippet file not being found (build failing with `SnippetMissingError`).
So it seems currently only camelCase IDs are supported, but neither kebab-case nor snake_case.
Would be nice to add some more flexibility here.

### Benefits

For such IDs I usually prefer kebab-cased IDs for readability.

### Solution Idea

_No response_

Discussion:
facelessuser: @gir-bot lgtm

Patch:
diff --git a/docs/src/dictionary/en-custom.txt b/docs/src/dictionary/en-custom.txt
index 34bc1ed5..77322537 100644
--- a/docs/src/dictionary/en-custom.txt
+++ b/docs/src/dictionary/en-custom.txt
@@ -20,6 +20,8 @@ CommonMark
 CriticMarkup
 Ctrl
 DOM
+De
+Dedent
 Dicts
 Donath
 ElementTree
@@ -138,6 +140,8 @@ config
 configs
 coveragepy
 customizable
+de
+dedenting
 deprecations
 dev
 deviantart
diff --git a/docs/src/markdown/about/changelog.md b/docs/src/markdown/about/changelog.md
index cc7c71f2..25b51414 100644
--- a/docs/src/markdown/about/changelog.md
+++ b/docs/src/markdown/about/changelog.md
@@ -2,11 +2,16 @@
 
 ## 9.10
 
-- **NEW**: Blocks: Add new general purpose block extensions meant to be an alternative to: Admonitions, Details, Definition
-  Lists, and Tabbed. Also adds new HTML plugin for quick wrapping of content with arbitrary HTML elements. General
-  Purpose blocks are more of a new frame work to create block plugins using special fences.
+- **NEW**: Blocks: Add new experimental general purpose blocks that provide a framework for creating fenced block
+  containers for specialized parsing. A number of extensions utilizing general purpose blocks are included and are meant
+  to be an alternative to (and maybe one day replace): Admonitions, Details, Definition   Lists, and Tabbed. Also adds a
+  new HTML plugin for quick wrapping of content with arbitrary HTML elements.
 - **NEW**: Highlight: When enabling line spans and/or line anchors, if a code block has an ID associated with it, line
   ids will be generated using that code ID instead of the code block count.
+- **NEW**: Snippets: Expand section syntax to allow section names with `-` and `_`.
+- **NEW**: Snippets: When `check_paths` is enabled, and a specified section is not found, raise an error.
+- **NEW**: Snippets: Add new experimental feature `dedent_sections` that will de-indent (remove any common leading
+  whitespace from every line in text) from that block of text.
 
 ## 9.10b1
 
diff --git a/docs/src/markdown/extensions/snippets.md b/docs/src/markdown/extensions/snippets.md
index 0aff31d1..8c3e8137 100644
--- a/docs/src/markdown/extensions/snippets.md
+++ b/docs/src/markdown/extensions/snippets.md
@@ -219,6 +219,24 @@ snippets are not allowed to reference local snippet files.
 URL snippet support was introduced in 9.5.
 ///
 
+## Dedent Subsections
+
+/// new | New 9.10
+///
+
+/// warning | Experimental
+///
+
+By default, when a subsection is extracted from a file via the [section notation](#snippet-sections) or the
+[lines notation](#snippet-lines), the content is inserted exactly how it is extracted. Unfortunately, sometimes you are
+extracting an indented chunk, and you do not intend for that chunk to be indented.
+
+`dedent_subsections` is a recent option that has been added to see if it alleviates the issue. When specifying a
+subsection of a file to insert as a snippet, via "sections" or "lines", that content will have all common leading
+whitespace removed from every line in text.
+
+Depending on how the feature is received, it may be made the default in the future.
+
 ## Auto-Append Snippets
 
 Snippets is designed as a general way to target a file and inject it into a given Markdown file, but some times,
@@ -241,3 +259,4 @@ Option                 | Type            | Default          | Description
 `url_max_size`         | int             | `#!py3 33554432` | Sets an arbitrary max content size. If content length is reported to be larger, and exception will be thrown. Default is ~32 MiB.
 `url_timeout`          | float           | `#!py3 10.0`     | Passes an arbitrary timeout in seconds to URL requestor. By default this is set to 10 seconds.
 `url_request_headers`  | {string:string} | `#!py3 {}`       | Passes arbitrary headers to URL requestor. By default this is set to empty map.
+`dedent_subsections`   | bool            | `#!py3 False`    | Remove any common leading whitespace from every line in text of a subsection that is inserted via "sections" or by "lines".
diff --git a/pymdownx/snippets.py b/pymdownx/snippets.py
index cf5f903b..44318160 100644
--- a/pymdownx/snippets.py
+++ b/pymdownx/snippets.py
@@ -30,6 +30,7 @@ import re
 import codecs
 import os
 from . import util
+import textwrap
 
 MI = 1024 * 1024  # mebibyte (MiB)
 DEFAULT_URL_SIZE = MI * 32
@@ -67,12 +68,12 @@ class SnippetPreprocessor(Preprocessor):
         r'''(?xi)
         ^.*?
         (?P<inline_marker>-{1,}8<-{1,}[ \t]+)
-        \[[ \t]*(?P<type>start|end)[ \t]*:[ \t]*(?P<name>[a-z][0-9a-z]*)[ \t]*\]
+        \[[ \t]*(?P<type>start|end)[ \t]*:[ \t]*(?P<name>[a-z][-_0-9a-z]*)[ \t]*\]
         .*?$
         '''
     )
 
-    RE_SNIPPET_FILE = re.compile(r'(?i)(.*?)(?:(:[0-9]*)?(:[0-9]*)?|(:[a-z][0-9a-z]*)?)$')
+    RE_SNIPPET_FILE = re.compile(r'(?i)(.*?)(?:(:[0-9]*)?(:[0-9]*)?|(:[a-z][-_0-9a-z]*)?)$')
 
     def __init__(self, config, md):
         """Initialize."""
@@ -88,6 +89,7 @@ class SnippetPreprocessor(Preprocessor):
         self.url_max_size = config['url_max_size']
         self.url_timeout = config['url_timeout']
         self.url_request_headers = config['url_request_headers']
+        self.dedent_subsections = config['dedent_subsections']
         self.tab_length = md.tab_length
         super(SnippetPreprocessor, self).__init__()
 
@@ -96,6 +98,7 @@ class SnippetPreprocessor(Preprocessor):
 
         new_lines = []
         start = False
+        found = False
         for l in lines:
 
             # Found a snippet section marker with our specified name
@@ -105,6 +108,7 @@ class SnippetPreprocessor(Preprocessor):
                 # We found the start
                 if not start and m.group('type') == 'start':
                     start = True
+                    found = True
                     continue
 
                 # We found the end
@@ -114,13 +118,21 @@ class SnippetPreprocessor(Preprocessor):
 
                 # We found an end, but no start
                 else:
-                    return []
+                    break
 
             # We are currently in a section, so append the line
             if start:
                 new_lines.append(l)
 
-        return new_lines
+        if not found and self.check_paths:
+            raise SnippetMissingError("Snippet section '{}' could not be located".format(section))
+
+        return self.dedent(new_lines) if self.dedent_subsections else new_lines
+
+    def dedent(self, lines):
+        """De-indent lines."""
+
+        return textwrap.dedent('\n'.join(lines)).split('\n')
 
     def get_snippet_path(self, path):
         """Get snippet path."""
@@ -282,7 +294,7 @@ class SnippetPreprocessor(Preprocessor):
                             s_lines = [l.rstrip('\r\n') for l in f]
                             if start is not None or end is not None:
                                 s = slice(start, end)
-                                s_lines = s_lines[s]
+                                s_lines = self.dedent(s_lines[s]) if self.dedent_subsections else s_lines[s]
                             elif section:
                                 s_lines = self.extract_section(section, s_lines)
                     else:
@@ -291,7 +303,7 @@ class SnippetPreprocessor(Preprocessor):
                             s_lines = self.download(snippet)
                             if start is not None or end is not None:
                                 s = slice(start, end)
-                                s_lines = s_lines[s]
+                                s_lines = self.dedent(s_lines[s]) if self.dedent_subsections else s_lines[s]
                             elif section:
                                 s_lines = self.extract_section(section, s_lines)
                         except SnippetMissingError:
@@ -346,7 +358,8 @@ class SnippetExtension(Extension):
             'url_download': [False, "Download external URLs as snippets - Default: \"False\""],
             'url_max_size': [DEFAULT_URL_SIZE, "External URL max size (0 means no limit)- Default: 32 MiB"],
             'url_timeout': [DEFAULT_URL_TIMEOUT, 'Defualt URL timeout (0 means no timeout) - Default: 10 sec'],
-            'url_request_headers': [DEFAULT_URL_REQUEST_HEADERS, "Extra request Headers - Default: {}"]
+            'url_request_headers': [DEFAULT_URL_REQUEST_HEADERS, "Extra request Headers - Default: {}"],
+            'dedent_subsections': [False, "Dedent subsection extractions e.g. 'sections' and/or 'lines'."]
         }
 
         super(SnippetExtension, self).__init__(*args, **kwargs)


Test patch:
diff --git a/tests/test_extensions/_snippets/indented.txt b/tests/test_extensions/_snippets/indented.txt
new file mode 100644
index 00000000..b5628843
--- /dev/null
+++ b/tests/test_extensions/_snippets/indented.txt
@@ -0,0 +1,9 @@
+class SomeClass:
+    """Docstring."""
+
+    # --8<-- [start: py-section]
+    def some_method(self, param):
+        """Docstring."""
+
+        return param
+    # --8<-- [end: py-section]
diff --git a/tests/test_extensions/_snippets/section.txt b/tests/test_extensions/_snippets/section.txt
index ea43f9f4..ea99478c 100644
--- a/tests/test_extensions/_snippets/section.txt
+++ b/tests/test_extensions/_snippets/section.txt
@@ -1,19 +1,19 @@
-/* --8<-- [start: cssSection] */
+/* --8<-- [start: css-section] */
 div {
     color: red;
 }
-/* --8<-- [end: cssSection] */
+/* --8<-- [end: css-section] */
 
-<!-- -8<- [start: htmlSection] -->
+<!-- -8<- [start: html-section] -->
 <div><p>content</p></div>
-<!-- -8<- [end: htmlSection] -->
+<!-- -8<- [end: html-section] -->
 
-/* --8<-- [end: cssSection2] */
-/* --8<-- [start: cssSection2] */
+/* --8<-- [end: css-section2] */
+/* --8<-- [start: css-section2] */
 div {
     color: red;
 }
-/* --8<-- [end: cssSection2] */
+/* --8<-- [end: css-section2] */
 
-<!-- --8<-- [start: htmlSection2] -->
+<!-- --8<-- [start: html-section2] -->
 <div><p>content</p></div>
diff --git a/tests/test_extensions/test_snippets.py b/tests/test_extensions/test_snippets.py
index 4bb03d90..31006020 100644
--- a/tests/test_extensions/test_snippets.py
+++ b/tests/test_extensions/test_snippets.py
@@ -7,6 +7,79 @@ from unittest.mock import patch, MagicMock
 BASE = os.path.abspath(os.path.dirname(__file__))
 
 
+class TestSnippetDedent(util.MdCase):
+    """Test snippet cases."""
+
+    extension = [
+        'pymdownx.snippets', 'pymdownx.superfences'
+    ]
+
+    extension_configs = {
+        'pymdownx.snippets': {
+            'base_path': [os.path.join(BASE, '_snippets')],
+            'dedent_subsections': True
+        }
+    }
+
+    def test_dedent_section(self):
+        """Test dedenting sections."""
+
+        self.check_markdown(
+            R'''
+            ```text
+            ---8<--- "indented.txt:py-section"
+            ```
+            ''',  # noqa: W291
+            R'''
+            <div class="highlight"><pre><span></span><code>def some_method(self, param):
+                &quot;&quot;&quot;Docstring.&quot;&quot;&quot;
+
+                return param
+            </code></pre></div>
+            ''',
+            True
+        )
+
+    def test_dedent_lines(self):
+        """Test dedenting lines."""
+
+        self.check_markdown(
+            R'''
+            ```text
+            ---8<--- "indented.txt:5:8"
+            ```
+            ''',  # noqa: W291
+            R'''
+            <div class="highlight"><pre><span></span><code>def some_method(self, param):
+                &quot;&quot;&quot;Docstring.&quot;&quot;&quot;
+
+                return param
+            </code></pre></div>
+            ''',
+            True
+        )
+
+    def test_dedent_indented(self):
+        """Test dedenting sections that has indented insertion."""
+
+        self.check_markdown(
+            R'''
+            Paragraph
+
+                ---8<--- "indented.txt:py-section"
+            ''',  # noqa: W291
+            R'''
+            <p>Paragraph</p>
+            <pre><code>def some_method(self, param):
+                """Docstring."""
+
+                return param
+            </code></pre>
+            ''',
+            True
+        )
+
+
 class TestSnippets(util.MdCase):
     """Test snippet cases."""
 
@@ -229,7 +302,7 @@ class TestSnippets(util.MdCase):
         self.check_markdown(
             R'''
             ```
-            --8<-- "section.txt:cssSection"
+            --8<-- "section.txt:css-section"
             ```
             ''',
             '''
@@ -247,7 +320,7 @@ class TestSnippets(util.MdCase):
         self.check_markdown(
             R'''
             ```
-            -8<- "section.txt:cssSection"
+            -8<- "section.txt:css-section"
             ```
             ''',
             '''
@@ -265,7 +338,7 @@ class TestSnippets(util.MdCase):
         self.check_markdown(
             R'''
             --8<--
-            section.txt:htmlSection
+            section.txt:html-section
             --8<--
             ''',
             '''
@@ -280,7 +353,7 @@ class TestSnippets(util.MdCase):
         self.check_markdown(
             R'''
             -8<-
-            section.txt:htmlSection
+            section.txt:html-section
             -8<-
             ''',
             '''
@@ -295,7 +368,7 @@ class TestSnippets(util.MdCase):
         self.check_markdown(
             R'''
             --8<--
-            section.txt:cssSection2
+            section.txt:css-section2
             --8<--
             ''',
             '''
@@ -309,7 +382,7 @@ class TestSnippets(util.MdCase):
         self.check_markdown(
             R'''
             --8<--
-            section.txt:htmlSection2
+            section.txt:html-section2
             --8<--
             ''',
             '''
@@ -428,7 +501,7 @@ class TestSnippetsMissing(util.MdCase):
                 True
             )
 
-    def test_missing_lines(self):
+    def test_missing_file_lines(self):
         """Test missing file with line numbers."""
 
         with self.assertRaises(SnippetMissingError):
@@ -441,6 +514,19 @@ class TestSnippetsMissing(util.MdCase):
                 True
             )
 
+    def test_missing_section(self):
+        """Test missing section."""
+
+        with self.assertRaises(SnippetMissingError):
+            self.check_markdown(
+                R'''
+                --8<-- "section.txt:missing-section"
+                ''',
+                '''
+                ''',
+                True
+            )
+
 
 class TestSnippetsGracefulMissing(util.MdCase):
     """Test snippet file case."""
@@ -731,7 +817,7 @@ class TestURLSnippets(util.MdCase):
 
         self.check_markdown(
             R'''
-            --8<-- "https://test.com/myfile.md:htmlSection"
+            --8<-- "https://test.com/myfile.md:html-section"
             ''',
             '''
             <div><p>content</p></div>
@@ -740,6 +826,57 @@ class TestURLSnippets(util.MdCase):
         )
 
 
+class TestURLDedentSnippets(util.MdCase):
+    """Test snippet URL cases."""
+
+    extension = [
+        'pymdownx.snippets', 'pymdownx.superfences'
+    ]
+
+    extension_configs = {
+        'pymdownx.snippets': {
+            'base_path': [os.path.join(BASE, '_snippets')],
+            'url_download': True,
+            'dedent_subsections': True
+        }
+    }
+
+    @patch('urllib.request.urlopen')
+    def test_url_sections(self, mock_urlopen):
+        """Test specifying a section in a URL."""
+
+        content = []
+        length = 0
+        with open('tests/test_extensions/_snippets/indented.txt', 'rb') as f:
+            for l in f:
+                length += len(l)
+                content.append(l)
+
+        cm = MagicMock()
+        cm.status = 200
+        cm.code = 200
+        cm.readlines.return_value = content
+        cm.headers = {'content-length': length}
+        cm.__enter__.return_value = cm
+        mock_urlopen.return_value = cm
+
+        self.check_markdown(
+            R'''
+            ```
+            --8<-- "https://test.com/myfile.md:py-section"
+            ```
+            ''',
+            '''
+            <div class="highlight"><pre><span></span><code>def some_method(self, param):
+                &quot;&quot;&quot;Docstring.&quot;&quot;&quot;
+
+                return param
+            </code></pre></div>
+            ''',
+            True
+        )
+
+
 class TestURLSnippetsNoMax(util.MdCase):
     """Test snippet URL cases no max size."""