Repository: executablebooks/MyST-Parser
Pull request #777 (2023-06-06 09:37:07)

👌 IMPROVE: Substitution extension
Allow any value type (including dict, list, datetime) and emit a `myst.substitution` warning for errors in resolving the substitution.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) Report
Patch coverage: **`100.00`**% and project coverage change: **`+0.23`** :tada:
> Comparison is base [(`b043351`)](https://app.codecov.io/gh/executablebooks/MyST-Parser/commit/b043351c6831a6a4eacbd742b7f795a1c4b71710?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) 90.00% compared to head [(`e639ce7`)](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) 90.23%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master     #777      +/-   ##
==========================================
+ Coverage   90.00%   90.23%   +0.23%     
==========================================
  Files          23       23              
  Lines        2970     2970              
==========================================
+ Hits         2673     2680       +7     
+ Misses        297      290       -7     
```

| Flag | Coverage Δ | |
|---|---|---|
| pytests | `90.23% <100.00%> (+0.23%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) | Coverage Δ | |
|---|---|---|
| [myst\_parser/config/dc\_validators.py](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvY29uZmlnL2RjX3ZhbGlkYXRvcnMucHk=) | `83.63% <100.00%> (+0.30%)` | :arrow_up: |
| [myst\_parser/config/main.py](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvY29uZmlnL21haW4ucHk=) | `86.26% <100.00%> (ø)` | |
| [myst\_parser/mdit\_to\_docutils/base.py](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvbWRpdF90b19kb2N1dGlscy9iYXNlLnB5) | `93.41% <100.00%> (+0.72%)` | :arrow_up: |
| [myst\_parser/warnings\_.py](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvd2FybmluZ3NfLnB5) | `96.22% <100.00%> (+0.07%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/executablebooks/MyST-Parser/pull/777?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks).


Patch:
diff --git a/myst_parser/config/dc_validators.py b/myst_parser/config/dc_validators.py
index 4f27130..4e13292 100644
--- a/myst_parser/config/dc_validators.py
+++ b/myst_parser/config/dc_validators.py
@@ -41,6 +41,12 @@ class ValidatorType(Protocol):
         ...
 
 
+def any_(inst, field, value, suffix=""):
+    """
+    A validator that does not perform any validation.
+    """
+
+
 def instance_of(type_: type[Any] | tuple[type[Any], ...]) -> ValidatorType:
     """
     A validator that raises a `TypeError` if the initializer is called
diff --git a/myst_parser/config/main.py b/myst_parser/config/main.py
index 8361843..ac5fa83 100644
--- a/myst_parser/config/main.py
+++ b/myst_parser/config/main.py
@@ -18,6 +18,7 @@ from typing import (
 
 from myst_parser.warnings_ import MystWarnings
 from .dc_validators import (
+    any_,
     deep_iterable,
     deep_mapping,
     in_,
@@ -335,12 +336,10 @@ class MdParserConfig:
 
     # Extension specific
 
-    substitutions: Dict[str, Union[str, int, float]] = dc.field(
+    substitutions: Dict[str, Any] = dc.field(
         default_factory=dict,
         metadata={
-            "validator": deep_mapping(
-                instance_of(str), instance_of((str, int, float)), instance_of(dict)
-            ),
+            "validator": deep_mapping(instance_of(str), any_, instance_of(dict)),
             "merge_topmatter": True,
             "help": "Substitutions mapping",
             "extension": "substitutions",
diff --git a/myst_parser/mdit_to_docutils/base.py b/myst_parser/mdit_to_docutils/base.py
index 25a010c..c70ae8f 100644
--- a/myst_parser/mdit_to_docutils/base.py
+++ b/myst_parser/mdit_to_docutils/base.py
@@ -1854,11 +1854,12 @@ class DocutilsRenderer(RendererProtocol):
                 variable_context
             )
         except Exception as error:
-            error_msg = self.reporter.error(
+            self.create_warning(
                 f"Substitution error:{error.__class__.__name__}: {error}",
+                MystWarnings.SUBSTITUTION,
                 line=position,
+                append_to=self.current_node,
             )
-            self.current_node += [error_msg]
             return
 
         # handle circular references
@@ -1869,11 +1870,12 @@ class DocutilsRenderer(RendererProtocol):
         self.document.sub_references = getattr(self.document, "sub_references", set())
         cyclic = references.intersection(self.document.sub_references)
         if cyclic:
-            error_msg = self.reporter.error(
+            self.create_warning(
                 f"circular substitution reference: {cyclic}",
+                MystWarnings.SUBSTITUTION,
                 line=position,
+                append_to=self.current_node,
             )
-            self.current_node += [error_msg]
             return
 
         # TODO improve error reporting;
diff --git a/myst_parser/warnings_.py b/myst_parser/warnings_.py
index fc54024..7493d89 100644
--- a/myst_parser/warnings_.py
+++ b/myst_parser/warnings_.py
@@ -59,6 +59,8 @@ class MystWarnings(Enum):
     """HTML could not be parsed."""
     INVALID_ATTRIBUTE = "attribute"
     """Invalid attribute value."""
+    SUBSTITUTION = "substitution"
+    """Substitution could not be resolved."""
 
 
 def _is_suppressed_warning(


Test patch:
diff --git a/tests/test_renderers/fixtures/myst-config.txt b/tests/test_renderers/fixtures/myst-config.txt
index 86e36d7..e0e6930 100644
--- a/tests/test_renderers/fixtures/myst-config.txt
+++ b/tests/test_renderers/fixtures/myst-config.txt
@@ -234,15 +234,26 @@ text
         text
 .
 
-[substitutions] --myst-enable-extensions=substitution --myst-substitutions='{"a": "b", "c": "d"}'
+[substitutions] --myst-enable-extensions=substitution --myst-substitutions='{"a": "b", "c": "d", "e": "{{f}}", "f": "{{e}}"}'
 .
-{{a}} {{c}}
+{{a}} {{c}} {{x}} {{e}}
 .
 <document source="<string>">
     <paragraph>
         b
 
         d
+
+        <system_message level="2" line="1" source="<string>" type="WARNING">
+            <paragraph>
+                Substitution error:UndefinedError: 'x' is undefined [myst.substitution]
+
+        <system_message level="2" line="3" source="<string>" type="WARNING">
+            <paragraph>
+                circular substitution reference: {'e'} [myst.substitution]
+
+<string>:1: (WARNING/2) Substitution error:UndefinedError: 'x' is undefined [myst.substitution]
+<string>:3: (WARNING/2) circular substitution reference: {'e'} [myst.substitution]
 .
 
 [attrs_image] --myst-enable-extensions=attrs_image
diff --git a/tests/test_renderers/fixtures/reporter_warnings.md b/tests/test_renderers/fixtures/reporter_warnings.md
index c5b26ab..776d0e8 100644
--- a/tests/test_renderers/fixtures/reporter_warnings.md
+++ b/tests/test_renderers/fixtures/reporter_warnings.md
@@ -62,7 +62,6 @@ myst:
 .
 <string>:1: (WARNING/2) 'title_to_header' must be of type <class 'bool'> (got 1 that is a <class 'int'>). [myst.topmatter]
 <string>:1: (WARNING/2) 'url_schemes' is not a list of strings: [1] [myst.topmatter]
-<string>:1: (WARNING/2) 'substitutions['key']' must be of type (<class 'str'>, <class 'int'>, <class 'float'>) (got [] that is a <class 'list'>). [myst.topmatter]
 .
 
 Bad HTML Meta
diff --git a/tests/test_sphinx/sourcedirs/substitutions/index.md b/tests/test_sphinx/sourcedirs/substitutions/index.md
index e1dd672..9773a95 100644
--- a/tests/test_sphinx/sourcedirs/substitutions/index.md
+++ b/tests/test_sphinx/sourcedirs/substitutions/index.md
@@ -18,6 +18,11 @@ myst:
       Inline note
       ```
     override: Overridden by front matter
+    date: 2020-01-01
+    nested_list:
+      - item1
+    nested_dict:
+      key1: value1
 
 ---
 
@@ -54,3 +59,9 @@ Using env and filters:
 ```{toctree}
 other.md
 ```
+
+{{ date.strftime("%b %d, %Y") }}
+
+{{ nested_list.0 }}
+
+{{ nested_dict.key1 }}
diff --git a/tests/test_sphinx/test_sphinx_builds/test_substitutions.html b/tests/test_sphinx/test_sphinx_builds/test_substitutions.html
index c5b3f43..38ff9ba 100644
--- a/tests/test_sphinx/test_sphinx_builds/test_substitutions.html
+++ b/tests/test_sphinx/test_sphinx_builds/test_substitutions.html
@@ -77,6 +77,15 @@
      </li>
     </ul>
    </div>
+   <p>
+    Jan 01, 2020
+   </p>
+   <p>
+    item1
+   </p>
+   <p>
+    value1
+   </p>
   </div>
  </div>
 </div>
diff --git a/tests/test_sphinx/test_sphinx_builds/test_substitutions.xml b/tests/test_sphinx/test_sphinx_builds/test_substitutions.xml
index 9a63ee2..37bd766 100644
--- a/tests/test_sphinx/test_sphinx_builds/test_substitutions.xml
+++ b/tests/test_sphinx/test_sphinx_builds/test_substitutions.xml
@@ -53,3 +53,9 @@
         INDEX
     <compound classes="toctree-wrapper">
         <toctree caption="True" entries="(None,\ 'other')" glob="False" hidden="False" includefiles="other" includehidden="False" maxdepth="-1" numbered="0" parent="index" rawentries="" titlesonly="False">
+    <paragraph>
+        Jan 01, 2020
+    <paragraph>
+        item1
+    <paragraph>
+        value1