Repository: executablebooks/MyST-Parser
Pull request #490 (2022-01-04 17:33:20)

🐛 FIX: Source line reporting for nested parsing
Previously, nested parsing of content within directives
would not correctly propagate the starting line number
of the content.

For example:

````markdown
Paragraph

```{note}
:class: abc
:name: xyz

{unknown}`a`
```
````

Would incorrectly report the error on line 1:

```
<string>:1: (ERROR/3) Unknown interpreted text role "unknown".
```

but now correctly reports on line 7:

```
<string>:7: (ERROR/3) Unknown interpreted text role "unknown".
```


codecov[bot]: # [Codecov](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) Report
> Merging [#490](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) (06e409d) into [master](https://codecov.io/gh/executablebooks/MyST-Parser/commit/0f2902ab5eb6236805ed24dc387b34c00fe41744?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) (0f2902a) will **decrease** coverage by `0.26%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490/graphs/tree.svg?width=650&height=150&src=pr&token=ggpNnuB7ll&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks)](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks)

```diff
@@            Coverage Diff             @@
##           master     #490      +/-   ##
==========================================
- Coverage   89.86%   89.60%   -0.27%     
==========================================
  Files          16       16              
  Lines        2062     2068       +6     
==========================================
  Hits         1853     1853              
- Misses        209      215       +6     
```

| Flag | Coverage Δ | |
|---|---|---|
| pytests | `89.60% <100.00%> (-0.27%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks) | Coverage Δ | |
|---|---|---|
| [myst\_parser/docutils\_renderer.py](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvZG9jdXRpbHNfcmVuZGVyZXIucHk=) | `91.66% <100.00%> (+0.03%)` | :arrow_up: |
| [myst\_parser/mocking.py](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvbW9ja2luZy5weQ==) | `86.59% <100.00%> (ø)` | |
| [myst\_parser/parse\_directives.py](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvcGFyc2VfZGlyZWN0aXZlcy5weQ==) | `87.64% <100.00%> (+0.43%)` | :arrow_up: |
| [myst\_parser/mathjax.py](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks#diff-bXlzdF9wYXJzZXIvbWF0aGpheC5weQ==) | `82.60% <0.00%> (-13.05%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks). Last update [0f2902a...06e409d](https://codecov.io/gh/executablebooks/MyST-Parser/pull/490?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=executablebooks).


Patch:
diff --git a/myst_parser/docutils_renderer.py b/myst_parser/docutils_renderer.py
index 0207ac2..9caf091 100644
--- a/myst_parser/docutils_renderer.py
+++ b/myst_parser/docutils_renderer.py
@@ -262,10 +262,19 @@ class DocutilsRenderer(RendererProtocol):
             self.document.note_substitution_def(substitution_node, f"wordcount-{key}")
 
     def nested_render_text(self, text: str, lineno: int) -> None:
-        """Render unparsed text."""
+        """Render unparsed text.
+
+        :param text: the text to render
+        :param lineno: the starting line number of the text, within the full source
+        """
 
         tokens = self.md.parse(text + "\n", self.md_env)
 
+        # update the line numbers
+        for token in tokens:
+            if token.map:
+                token.map = [token.map[0] + lineno, token.map[1] + lineno]
+
         # remove front matter
         if tokens and tokens[0].type == "front_matter":
             tokens.pop(0)
@@ -1139,7 +1148,7 @@ class DocutilsRenderer(RendererProtocol):
             directive_class.option_spec["relative-docs"] = directives.path
 
         try:
-            arguments, options, body_lines = parse_directive_text(
+            arguments, options, body_lines, content_offset = parse_directive_text(
                 directive_class, first_line, content
             )
         except DirectiveParsingError as error:
@@ -1175,7 +1184,7 @@ class DocutilsRenderer(RendererProtocol):
                 # the absolute line number of the first line of the directive
                 lineno=position,
                 # the line offset of the first line of the content
-                content_offset=0,  # TODO get content offset from `parse_directive_text`
+                content_offset=content_offset,
                 # a string containing the entire directive
                 block_text="\n".join(body_lines),
                 state=state,
diff --git a/myst_parser/mocking.py b/myst_parser/mocking.py
index 382596e..67ad419 100644
--- a/myst_parser/mocking.py
+++ b/myst_parser/mocking.py
@@ -114,14 +114,14 @@ class MockState:
         if option_presets:
             raise MockingError("parse_directive_block: option_presets not implemented")
         # TODO should argument_str always be ""?
-        arguments, options, body_lines = parse_directive_text(
+        arguments, options, body_lines, content_offset = parse_directive_text(
             directive, "", "\n".join(content)
         )
         return (
             arguments,
             options,
             StringList(body_lines, source=content.source),
-            line_offset + len(content) - len(body_lines),
+            line_offset + content_offset,
         )
 
     def nested_parse(
diff --git a/myst_parser/parse_directives.py b/myst_parser/parse_directives.py
index 64d0e38..b053dea 100644
--- a/myst_parser/parse_directives.py
+++ b/myst_parser/parse_directives.py
@@ -36,7 +36,7 @@ This is to allow for separation between the option block and content.
 import datetime
 import re
 from textwrap import dedent
-from typing import Any, Callable, Dict, Type
+from typing import Any, Callable, Dict, List, Tuple, Type
 
 import yaml
 from docutils.parsers.rst import Directive
@@ -54,7 +54,7 @@ def parse_directive_text(
     first_line: str,
     content: str,
     validate_options: bool = True,
-):
+) -> Tuple[List[str], dict, List[str], int]:
     """Parse (and validate) the full directive text.
 
     :param first_line: The text on the same line as the directive name.
@@ -62,17 +62,19 @@ def parse_directive_text(
     :param content: All text after the first line. Can include options.
     :param validate_options: Whether to validate the values of options
 
+    :returns: (arguments, options, body_lines, content_offset)
     """
     if directive_class.option_spec:
         body, options = parse_directive_options(
             content, directive_class, validate=validate_options
         )
+        body_lines = body.splitlines()
+        content_offset = len(content.splitlines()) - len(body_lines)
     else:
         # If there are no possible options, we do not look for a YAML block
         options = {}
-        body = content
-
-    body_lines = body.splitlines()
+        body_lines = content.splitlines()
+        content_offset = 0
 
     if not (
         directive_class.required_arguments
@@ -91,12 +93,13 @@ def parse_directive_text(
     # this is to allow space between the options and the content
     if body_lines and not body_lines[0].strip():
         body_lines = body_lines[1:]
+        content_offset += 1
 
     # check for body content
     if body_lines and not directive_class.has_content:
         raise DirectiveParsingError("No content permitted")
 
-    return arguments, options, body_lines
+    return arguments, options, body_lines, content_offset
 
 
 def parse_directive_options(


Test patch:
diff --git a/tests/test_renderers/fixtures/mock_include_errors.md b/tests/test_renderers/fixtures/mock_include_errors.md
index de35edd..1e5c2e9 100644
--- a/tests/test_renderers/fixtures/mock_include_errors.md
+++ b/tests/test_renderers/fixtures/mock_include_errors.md
@@ -20,5 +20,5 @@ Error in include file:
 ```{include} bad.md
 ```
 .
-tmpdir/bad.md:1: (ERROR/3) Unknown interpreted text role "a".
+tmpdir/bad.md:2: (ERROR/3) Unknown interpreted text role "a".
 .
diff --git a/tests/test_renderers/fixtures/reporter_warnings.md b/tests/test_renderers/fixtures/reporter_warnings.md
index bda330d..2fd6d67 100644
--- a/tests/test_renderers/fixtures/reporter_warnings.md
+++ b/tests/test_renderers/fixtures/reporter_warnings.md
@@ -136,5 +136,19 @@ header nested in admonition
 # Header
 ```
 .
-<string>:1: (WARNING/2) Header nested in this element can lead to unexpected outcomes
+<string>:2: (WARNING/2) Header nested in this element can lead to unexpected outcomes
+.
+
+nested parse warning
+.
+Paragraph
+
+```{note}
+:class: abc
+:name: xyz
+
+{unknown}`a`
+```
+.
+<string>:7: (ERROR/3) Unknown interpreted text role "unknown".
 .
diff --git a/tests/test_renderers/test_parse_directives.py b/tests/test_renderers/test_parse_directives.py
index 07e4c0f..c85e663 100644
--- a/tests/test_renderers/test_parse_directives.py
+++ b/tests/test_renderers/test_parse_directives.py
@@ -6,11 +6,21 @@ from docutils.parsers.rst.directives.body import Rubric
 from myst_parser.parse_directives import DirectiveParsingError, parse_directive_text
 
 
-@pytest.mark.parametrize("klass,arguments,content", [(Note, "", "a"), (Note, "a", "")])
+@pytest.mark.parametrize(
+    "klass,arguments,content",
+    [(Note, "", "a"), (Note, "a", ""), (Note, "", ":class: name\n\na")],
+)
 def test_parsing(klass, arguments, content, data_regression):
-    arguments, options, body_lines = parse_directive_text(klass, arguments, content)
+    arguments, options, body_lines, content_offset = parse_directive_text(
+        klass, arguments, content
+    )
     data_regression.check(
-        {"arguments": arguments, "options": options, "body": body_lines}
+        {
+            "arguments": arguments,
+            "options": options,
+            "body": body_lines,
+            "content_offset": content_offset,
+        }
     )
 
 
diff --git a/tests/test_renderers/test_parse_directives/test_parsing_Note___class__name_n_na_.yml b/tests/test_renderers/test_parse_directives/test_parsing_Note___class__name_n_na_.yml
new file mode 100644
index 0000000..8b9bbdf
--- /dev/null
+++ b/tests/test_renderers/test_parse_directives/test_parsing_Note___class__name_n_na_.yml
@@ -0,0 +1,7 @@
+arguments: []
+body:
+- a
+content_offset: 2
+options:
+  class:
+  - name
diff --git a/tests/test_renderers/test_parse_directives/test_parsing_Note__a_.yml b/tests/test_renderers/test_parse_directives/test_parsing_Note__a_.yml
index f4a5517..fd1e558 100644
--- a/tests/test_renderers/test_parse_directives/test_parsing_Note__a_.yml
+++ b/tests/test_renderers/test_parse_directives/test_parsing_Note__a_.yml
@@ -1,4 +1,5 @@
 arguments: []
 body:
 - a
+content_offset: 0
 options: {}
diff --git a/tests/test_renderers/test_parse_directives/test_parsing_Note_a__.yml b/tests/test_renderers/test_parse_directives/test_parsing_Note_a__.yml
index f4a5517..fd1e558 100644
--- a/tests/test_renderers/test_parse_directives/test_parsing_Note_a__.yml
+++ b/tests/test_renderers/test_parse_directives/test_parsing_Note_a__.yml
@@ -1,4 +1,5 @@
 arguments: []
 body:
 - a
+content_offset: 0
 options: {}