Repository: mwouts/jupytext
Pull request #473 (2020-03-29 09:18:35)

Update parsing of myst notebooks to new (markdown-it based) parser
Hey @mwouts, first just to clarify this should not be merged just yet. It is an update to use the (currently) pre-release ExecutableBookProject/MyST-Parser#123, which utilises [markdown-it-py](https://github.com/ExecutableBookProject/markdown-it-py), my shiny new markdown parser 😁 

As you can see, it does not affect any of the tests, and thus shouldn't be noticeable on the front-end.

codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/473?src=pr&el=h1) Report
> Merging [#473](https://codecov.io/gh/mwouts/jupytext/pull/473?src=pr&el=desc) into [master](https://codecov.io/gh/mwouts/jupytext/commit/c2b3bfb5ff7829495b3c65381abbced5048075ec&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `97.02%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/473/graphs/tree.svg?width=650&height=150&src=pr&token=a1pNjsiILi)](https://codecov.io/gh/mwouts/jupytext/pull/473?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #473   +/-   ##
=======================================
  Coverage   98.86%   98.86%           
=======================================
  Files          80       80           
  Lines        7899     7904    +5     
=======================================
+ Hits         7809     7814    +5     
  Misses         90       90           
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/473?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jupytext/myst.py](https://codecov.io/gh/mwouts/jupytext/pull/473/diff?src=pr&el=tree#diff-anVweXRleHQvbXlzdC5weQ==) | `98.36% <96.93%> (+0.02%)` | :arrow_up: |
| [tests/test\_ipynb\_to\_myst.py](https://codecov.io/gh/mwouts/jupytext/pull/473/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9pcHluYl90b19teXN0LnB5) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/473?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/473?src=pr&el=footer). Last update [c2b3bfb...39a262f](https://codecov.io/gh/mwouts/jupytext/pull/473?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

lgtm-com[bot]: This pull request **fixes 1 alert** when merging 8a8fb3da22eafa330e6952acf65731e233a717d1 into c2b3bfb5ff7829495b3c65381abbced5048075ec - [view on LGTM.com](https://lgtm.com/projects/g/mwouts/jupytext/rev/pr-6de967ed08d262f481a11bb7610bfe1ea2e52597)

**fixed alerts:**

* 1 for Module is imported with &#39;import&#39; and &#39;import from&#39;

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f22230eb..a06b0ad4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,7 @@
 
 **Changed**
 - Use `os.path.samefile` when searching for the kernel that corresponds to the current environment (`--set-kernel -`)
+- The outputs from the `.ipynb` file are matched with the input cells from the text file with less strict rules. In this version, a search and replace on the text file will not remove the outputs any more (#464).
 
 **Fixed**
 - Fixed the CLI example for not commenting out magic commands: `--opt comment_magics=false`. In addition, most of the `jupytext` commands in `using-cli.md` are now tested! (#465)
diff --git a/jupytext/combine.py b/jupytext/combine.py
index c30751ae..c77d0c9b 100644
--- a/jupytext/combine.py
+++ b/jupytext/combine.py
@@ -19,18 +19,15 @@ def black_invariant(text, chars=None):
     return text
 
 
-def same_content(ref, test):
+def same_content(ref, test, endswith=False):
     """Is the content of two cells the same, up to reformating by black"""
+    if endswith:
+        return black_invariant(ref).endswith(black_invariant(test))
     return black_invariant(ref) == black_invariant(test)
 
 
 def combine_inputs_with_outputs(nb_source, nb_outputs, fmt=None):
-    """Copy outputs and metadata of the second notebook into
-    the first one, for cells that have matching inputs"""
-
-    output_code_cells = [cell for cell in nb_outputs.cells if cell.cell_type == 'code']
-    output_other_cells = [cell for cell in nb_outputs.cells if cell.cell_type != 'code']
-
+    """Copy outputs and metadata of the second notebook into the first one."""
     # nbformat version number taken from the notebook with outputs
     assert nb_outputs.nbformat == nb_source.nbformat, \
         "The notebook with outputs is in format {}.{}, please upgrade it to {}.x".format(
@@ -60,33 +57,89 @@ def combine_inputs_with_outputs(nb_source, nb_outputs, fmt=None):
     else:
         cell_metadata_filter = nb_source.metadata.get('jupytext', {}).get('cell_metadata_filter')
 
-    for cell in nb_source.cells:
+    outputs_map = map_outputs_to_inputs(nb_source.cells, nb_outputs.cells)
+
+    for cell, j in zip(nb_source.cells, outputs_map):
         # Remove outputs to warranty that trust of returned notebook is that of second notebook
         if cell.cell_type == 'code':
             cell.execution_count = None
             cell.outputs = []
 
-            for i, ocell in enumerate(output_code_cells):
-                if same_content(cell.source, ocell.source):
-                    # Fill outputs with that of second notebook
-                    cell.execution_count = ocell.execution_count
-                    cell.outputs = ocell.outputs
-
-                    # Restore the filtered output cell metadata
-                    cell.metadata = restore_filtered_metadata(cell.metadata, ocell.metadata,
-                                                              cell_metadata_filter, _IGNORE_CELL_METADATA)
+            if j is not None:
+                ocell = nb_outputs.cells[j]
+                cell.execution_count = ocell.execution_count
+                cell.outputs = ocell.outputs
 
-                    output_code_cells = output_code_cells[(i + 1):]
-                    break
+                # Restore the filtered output cell metadata
+                cell.metadata = restore_filtered_metadata(cell.metadata, ocell.metadata,
+                                                          cell_metadata_filter, _IGNORE_CELL_METADATA)
         else:
-            for i, ocell in enumerate(output_other_cells):
-                if cell.cell_type == ocell.cell_type and same_content(cell.source, ocell.source):
-                    # The 'spin' format does not allow metadata on non-code cells
-                    cell.metadata = restore_filtered_metadata(cell.metadata, ocell.metadata,
-                                                              '-all' if format_name == 'spin' else cell_metadata_filter,
-                                                              _IGNORE_CELL_METADATA)
-
-                    output_other_cells = output_other_cells[(i + 1):]
-                    break
+            if j is not None:
+                ocell = nb_outputs.cells[j]
+
+                # The 'spin' format does not allow metadata on non-code cells
+                cell.metadata = restore_filtered_metadata(cell.metadata, ocell.metadata,
+                                                          '-all' if format_name == 'spin' else cell_metadata_filter,
+                                                          _IGNORE_CELL_METADATA)
 
     return nb_source
+
+
+def map_outputs_to_inputs(cells_inputs, cells_outputs):
+    """Returns a map i->(j or None) that maps the cells with outputs to the input cells"""
+    n_in = len(cells_inputs)
+    n_out = len(cells_outputs)
+    outputs_map = [None] * n_in
+
+    # First rule: match based on cell type, content, in increasing order
+    current_j = 0
+    for i in range(n_in):
+        cell_input = cells_inputs[i]
+        for j in range(current_j, n_out):
+            cell_output = cells_outputs[j]
+            if cell_input.cell_type == cell_output.cell_type and same_content(cell_input.source, cell_output.source):
+                outputs_map[i] = j
+                current_j = j + 1
+                break
+
+    # Second rule: match unused outputs based on cell type and content
+    # Third rule: is the new cell the final part of a previous cell with outputs?
+    unused_ouputs = set(range(n_out)).difference(outputs_map)
+    for endswith in [False, True]:
+        if not unused_ouputs:
+            return outputs_map
+
+        for i in range(n_in):
+            if outputs_map[i] is not None:
+                continue
+            cell_input = cells_inputs[i]
+            for j in unused_ouputs:
+                cell_output = cells_outputs[j]
+                if cell_input.cell_type == cell_output.cell_type and \
+                        same_content(cell_output.source, cell_input.source, endswith):
+                    outputs_map[i] = j
+                    unused_ouputs.remove(j)
+                    break
+
+    # Fourth rule: match based on increasing index, for non-empty cells
+    if not unused_ouputs:
+        return outputs_map
+
+    prev_j = -1
+    for i in range(n_in):
+        if outputs_map[i] is not None:
+            prev_j = outputs_map[i]
+            continue
+
+        j = prev_j + 1
+        if j not in unused_ouputs:
+            continue
+
+        cell_input = cells_inputs[i]
+        cell_output = cells_outputs[j]
+        if cell_input.cell_type == cell_output.cell_type and cell_input.source:
+            outputs_map[i] = j
+            unused_ouputs.remove(j)
+            prev_j = j
+
+    return outputs_map
diff --git a/jupytext/formats.py b/jupytext/formats.py
index 7ddc85f4..f7fa3c4d 100644
--- a/jupytext/formats.py
+++ b/jupytext/formats.py
@@ -232,7 +232,7 @@ def read_format_from_metadata(text, ext):
 
 def guess_format(text, ext):
     """Guess the format and format options of the file, given its extension and content"""
-    if matches_mystnb(text, ext):
+    if is_myst_available() and matches_mystnb(text, ext):
         return MYST_FORMAT_NAME, {}
     lines = text.splitlines()
 
diff --git a/jupytext/myst.py b/jupytext/myst.py
index 682e66d6..9da61514 100644
--- a/jupytext/myst.py
+++ b/jupytext/myst.py
@@ -3,21 +3,26 @@ This module contains round-trip conversion between
 myst formatted text documents and notebooks.
 """
 import json
+import warnings
 
 import nbformat as nbf
 import yaml
 
 MYST_FORMAT_NAME = "myst"
-CODE_DIRECTIVE = "code-cell"
-RAW_DIRECTIVE = "raw-cell"
+CODE_DIRECTIVE = "{code-cell}"
+RAW_DIRECTIVE = "{raw-cell}"
 
 
 def is_myst_available():
     """Whether the myst-parser package is available."""
     try:
-        import myst_parser  # noqa
+        from myst_parser import __version__  # noqa
     except ImportError:
         return False
+    major, minor = __version__.split(".")[:2]
+    if int(major) < 1 and int(minor) < 8:
+        warnings.warn("The installed myst-parser version is less than the required 0.8")
+        return False
     return True
 
 
@@ -39,51 +44,60 @@ def matches_mystnb(
     text,
     ext=None,
     requires_meta=True,
-    require_non_md=True,
-    return_nb=False,
-    store_line_numbers=False,
+    code_directive=CODE_DIRECTIVE,
+    raw_directive=RAW_DIRECTIVE,
 ):
-    """Attempt to distinguish a file as mystnb, only given its extension and content.
+    """Attempt to distinguish a file as myst, only given its extension and content.
 
     :param ext: the extension of the file
     :param requires_meta: requires the file to contain top matter metadata
-    :param require_non_md: whether to require that a non-markdown cell is present
-    :param return_nb: if a match is found, return the notebook
-    :param store_line_numbers: add a `_source_lines` key to cell metadata,
-        mapping to the source text.
+    :param code_directive: the name of the directive to search for containing code cells
+    :param raw_directive: the name of the directive to search for containing raw cells
     """
+    # is the extension uniquely associated with myst (i.e. not just .md)
     if ext and "." + ("." + ext).rsplit(".", 1)[1] in myst_extensions(no_md=True):
-        return (
-            myst_to_notebook(
-                text, ignore_bad_meta=True, store_line_numbers=store_line_numbers
-            )
-            if return_nb
-            else True
-        )
+        return True
+
+    # might the text contain metadata front matter
     if requires_meta and not text.startswith("---"):
         return False
-    try:
-        nb = myst_to_notebook(
-            text, ignore_bad_meta=True, store_line_numbers=store_line_numbers
-        )
-    except Exception:
-        return False
 
-    from jupytext.formats import format_name_for_ext
+    from myst_parser.main import default_parser
 
-    # Is the format information available in the jupytext text representation?
     try:
-        format_name = format_name_for_ext(nb.metadata, ext or ".md")
-    except AttributeError:
-        pass
-    else:
-        if format_name == MYST_FORMAT_NAME:
-            return nb if return_nb else True
-
-    if require_non_md and not any(c.cell_type != "markdown" for c in nb.cells):
+        # parse markdown file up to the block level (i.e. don't worry about inline text)
+        parser = default_parser("html", disable_syntax=["inline"])
+        tokens = parser.parse(text + "\n")
+    except Exception as err:
+        warnings.warn("myst-parse failed unexpectedly: {}".format(err))
         return False
 
-    return nb if return_nb else True
+    # Is the format information available in the jupytext text representation?
+    if tokens and tokens[0].type == "front_matter":
+        from jupytext.formats import format_name_for_ext
+
+        try:
+            metadata = yaml.safe_load(tokens[0].content)
+        except (yaml.parser.ParserError, yaml.scanner.ScannerError):
+            pass
+        else:
+            try:
+                format_name = format_name_for_ext(metadata, ext or ".md")
+            except AttributeError:
+                pass
+            else:
+                if format_name == MYST_FORMAT_NAME:
+                    return True
+
+    # is there at least on fenced code block with a code/raw directive language
+    for token in tokens:
+        if token.type == "fence" and (
+            token.info.startswith(code_directive)
+            or token.info.startswith(raw_directive)
+        ):
+            return True
+
+    return False
 
 
 class CompactDumper(yaml.SafeDumper):
@@ -144,190 +158,150 @@ class MystMetadataParsingError(Exception):
     """Error when parsing metadata from myst formatted text"""
 
 
-def _fmt_md(text):
+def strip_blank_lines(text):
     text = text.rstrip()
     while text and text.startswith("\n"):
         text = text[1:]
     return text
 
 
+def read_fenced_cell(token, cell_index, cell_type):
+    from myst_parser.parse_directives import DirectiveParsingError, parse_directive_text
+
+    try:
+        _, options, body_lines = parse_directive_text(
+            directive_class=MockDirective,
+            argument_str="",
+            content=token.content,
+            validate_options=False,
+        )
+    except DirectiveParsingError as err:
+        raise MystMetadataParsingError(
+            "{0} cell {1} at line {2} could not be read: {3}".format(
+                cell_type, cell_index, token.map[0] + 1, err
+            )
+        )
+    return options, body_lines
+
+
+def read_cell_metadata(token, cell_index):
+    metadata = {}
+    if token.content:
+        try:
+            metadata = json.loads(token.content.strip())
+        except Exception as err:
+            raise MystMetadataParsingError(
+                "Markdown cell {0} at line {1} could not be read: {2}".format(
+                    cell_index, token.map[0] + 1, err
+                )
+            )
+        if not isinstance(metadata, dict):
+            raise MystMetadataParsingError(
+                "Markdown cell {0} at line {1} is not a dict".format(
+                    cell_index, token.map[0] + 1
+                )
+            )
+
+    return metadata
+
+
 def myst_to_notebook(
     text,
     code_directive=CODE_DIRECTIVE,
     raw_directive=RAW_DIRECTIVE,
-    ignore_bad_meta=False,
-    store_line_numbers=False,
+    add_source_map=False,
 ):
     """Convert text written in the myst format to a notebook.
 
     :param text: the file text
     :param code_directive: the name of the directive to search for containing code cells
     :param raw_directive: the name of the directive to search for containing raw cells
-    :param ignore_bad_meta: ignore metadata that cannot be parsed as JSON/YAML
-    :param store_line_numbers: add a `_source_lines` key to cell metadata,
-        mapping to the source text.
+    :param add_source_map: add a `source_map` key to the notebook metadata,
+        which is a list of the starting source line number for each cell.
+
+    :raises MystMetadataParsingError if the metadata block is not valid JSON/YAML
 
     NOTE: we assume here that all of these directives are at the top-level,
     i.e. not nested in other directives.
     """
-    from mistletoe.base_elements import SourceLines
-    from mistletoe.parse_context import (
-        ParseContext,
-        get_parse_context,
-        set_parse_context,
-    )
-    from mistletoe.block_tokens import Document, CodeFence
-
-    from myst_parser.block_tokens import BlockBreak
-    from myst_parser.parse_directives import DirectiveParsingError, parse_directive_text
-    from myst_parser.docutils_renderer import DocutilsRenderer
+    from myst_parser.main import default_parser
+
+    # parse markdown file up to the block level (i.e. don't worry about inline text)
+    parser = default_parser("html", disable_syntax=["inline"])
+    tokens = parser.parse(text + "\n")
+    lines = text.splitlines()
+    md_start_line = 0
+
+    # get the document metadata
+    metadata_nb = {}
+    if tokens[0].type == "front_matter":
+        metadata = tokens.pop(0)
+        md_start_line = metadata.map[1]
+        try:
+            metadata_nb = yaml.safe_load(metadata.content)
+        except (yaml.parser.ParserError, yaml.scanner.ScannerError) as error:
+            raise MystMetadataParsingError("Notebook metadata: {}".format(error))
+
+    # create an empty notebook
+    nbf_version = nbf.v4
+    kwargs = {"metadata": nbf.from_dict(metadata_nb)}
+    notebook = nbf_version.new_notebook(**kwargs)
+    source_map = []  # this is a list of the starting line number for each cell
+
+    def _flush_markdown(start_line, token, md_metadata):
+        """When we find a cell we check if there is preceding text.o"""
+        endline = token.map[0] if token else len(lines)
+        md_source = strip_blank_lines("\n".join(lines[start_line:endline]))
+        meta = nbf.from_dict(md_metadata)
+        if md_source:
+            source_map.append(start_line)
+            notebook.cells.append(
+                nbf_version.new_markdown_cell(source=md_source, metadata=meta)
+            )
 
-    code_directive = "{{{0}}}".format(code_directive)
-    raw_directive = "{{{0}}}".format(raw_directive)
+    # iterate through the tokens to identify notebook cells
+    nesting_level = 0
+    md_metadata = {}
 
-    original_context = get_parse_context()
-    parse_context = ParseContext(
-        find_blocks=DocutilsRenderer.default_block_tokens,
-        find_spans=DocutilsRenderer.default_span_tokens,
-    )
+    for token in tokens:
 
-    if isinstance(text, SourceLines):
-        lines = text
-    else:
-        lines = SourceLines(text, standardize_ends=True)
+        nesting_level += token.nesting
 
-    try:
-        set_parse_context(parse_context)
-        doc = Document.read(lines, front_matter=True)
-        metadata_nb = {}
-        try:
-            metadata_nb = doc.front_matter.get_data() if doc.front_matter else {}
-        except (yaml.parser.ParserError, yaml.scanner.ScannerError) as error:
-            if not ignore_bad_meta:
-                raise MystMetadataParsingError("Notebook metadata: {}".format(error))
-        nbf_version = nbf.v4
-        kwargs = {"metadata": nbf.from_dict(metadata_nb)}
-        notebook = nbf_version.new_notebook(**kwargs)
-
-        current_line = 0 if not doc.front_matter else doc.front_matter.position.line_end
-        md_metadata = {}
-
-        for item in doc.walk(["CodeFence", "BlockBreak"]):
-            if isinstance(item.node, BlockBreak):
-                token = item.node  # type: BlockBreak
-                source = _fmt_md(
-                    "".join(lines.lines[current_line:token.position.line_start - 1])
-                )
-                if source:
-                    md_metadata = nbf.from_dict(md_metadata)
-                    if store_line_numbers:
-                        md_metadata["_source_lines"] = [
-                            current_line,
-                            token.position.line_start - 1,
-                        ]
-                    notebook.cells.append(
-                        nbf_version.new_markdown_cell(
-                            source=source, metadata=md_metadata,
-                        )
-                    )
-                if token.content:
-                    md_metadata = {}
-                    try:
-                        md_metadata = json.loads(token.content.strip())
-                    except Exception as err:
-                        if not ignore_bad_meta:
-                            raise MystMetadataParsingError(
-                                "markdown cell {0} at {1} could not be read: {2}".format(
-                                    len(notebook.cells) + 1, token.position, err
-                                )
-                            )
-                    if not isinstance(md_metadata, dict):
-                        if not ignore_bad_meta:
-                            raise MystMetadataParsingError(
-                                "markdown cell {0} at {1} is not a dict".format(
-                                    len(notebook.cells) + 1, token.position
-                                )
-                            )
-                else:
-                    md_metadata = {}
-                current_line = token.position.line_start
-            if isinstance(item.node, CodeFence) and item.node.language in [
-                code_directive,
-                raw_directive,
-            ]:
-                token = item.node  # type: CodeFence
-                # Note: we ignore anything after the directive on the first line
-                # this is reserved for the optional lexer name
-                # TODO: could log warning about if token.arguments != lexer name
-
-                options, body_lines = {}, []
-                try:
-                    _, options, body_lines = parse_directive_text(
-                        directive_class=MockDirective,
-                        argument_str="",
-                        content=token.children[0].content,
-                        validate_options=False,
-                    )
-                except DirectiveParsingError as err:
-                    if not ignore_bad_meta:
-                        raise MystMetadataParsingError(
-                            "Code cell {0} at {1} could not be read: {2}".format(
-                                len(notebook.cells) + 1, token.position, err
-                            )
-                        )
-
-                md_source = _fmt_md(
-                    "".join(lines.lines[current_line:token.position.line_start - 1])
-                )
-                if md_source:
-                    md_metadata = nbf.from_dict(md_metadata)
-                    if store_line_numbers:
-                        md_metadata["_source_lines"] = [
-                            current_line,
-                            token.position.line_start - 1,
-                        ]
-                    notebook.cells.append(
-                        nbf_version.new_markdown_cell(
-                            source=md_source, metadata=md_metadata,
-                        )
-                    )
-                current_line = token.position.line_end
-                md_metadata = {}
-
-                cell_metadata = nbf.from_dict(options)
-                if store_line_numbers:
-                    cell_metadata["_source_lines"] = [
-                        token.position.line_start,
-                        token.position.line_end,
-                    ]
-                if item.node.language == code_directive:
-                    notebook.cells.append(
-                        nbf_version.new_code_cell(
-                            source="\n".join(body_lines), metadata=cell_metadata,
-                        )
-                    )
-                if item.node.language == raw_directive:
-                    notebook.cells.append(
-                        nbf_version.new_raw_cell(
-                            source="\n".join(body_lines), metadata=cell_metadata,
-                        )
-                    )
-
-        # add the final markdown cell (if present)
-        if lines.lines[current_line:]:
-            md_metadata = nbf.from_dict(md_metadata)
-            if store_line_numbers:
-                md_metadata["_source_lines"] = [current_line, len(lines.lines)]
+        if nesting_level != 0:
+            # we ignore fenced block that are nested, e.g. as part of lists, etc
+            continue
+
+        if token.type == "fence" and token.info.startswith(code_directive):
+            _flush_markdown(md_start_line, token, md_metadata)
+            options, body_lines = read_fenced_cell(token, len(notebook.cells), "Code")
+            meta = nbf.from_dict(options)
+            source_map.append(token.map[0] + 1)
             notebook.cells.append(
-                nbf_version.new_markdown_cell(
-                    source=_fmt_md("".join(lines.lines[current_line:])),
-                    metadata=md_metadata,
-                )
+                nbf_version.new_code_cell(source="\n".join(body_lines), metadata=meta)
             )
+            md_metadata = {}
+            md_start_line = token.map[1]
+
+        elif token.type == "fence" and token.info.startswith(raw_directive):
+            _flush_markdown(md_start_line, token, md_metadata)
+            options, body_lines = read_fenced_cell(token, len(notebook.cells), "Raw")
+            meta = nbf.from_dict(options)
+            source_map.append(token.map[0] + 1)
+            notebook.cells.append(
+                nbf_version.new_raw_cell(source="\n".join(body_lines), metadata=meta)
+            )
+            md_metadata = {}
+            md_start_line = token.map[1]
+
+        elif token.type == "myst_block_break":
+            _flush_markdown(md_start_line, token, md_metadata)
+            md_metadata = read_cell_metadata(token, len(notebook.cells))
+            md_start_line = token.map[1]
 
-    finally:
-        set_parse_context(original_context)
+    _flush_markdown(md_start_line, None, md_metadata)
 
+    if add_source_map:
+        notebook.metadata["source_map"] = source_map
     return notebook
 
 
@@ -370,7 +344,7 @@ def notebook_to_myst(
             last_cell_md = True
 
         elif cell.cell_type in ["code", "raw"]:
-            string += "\n```{{{}}}".format(
+            string += "\n```{}".format(
                 code_directive if cell.cell_type == "code" else raw_directive
             )
             if pygments_lexer and cell.cell_type == "code":
diff --git a/jupytext/stringparser.py b/jupytext/stringparser.py
index 1b3c618d..bf38f7ee 100644
--- a/jupytext/stringparser.py
+++ b/jupytext/stringparser.py
@@ -48,16 +48,26 @@ class StringParser:
             if not self.python:
                 continue
 
-            if self.triple == char:
-                if line[i - 2:i + 1] == 3 * char and i >= self.triple_start + 3:
+            if line[i - 2:i + 1] == 3 * char and i >= self.triple_start + 3:
+                # End of a triple quote
+                if self.triple == char:
                     self.triple = None
+                    self.triple_start = i
                     continue
-            if self.triple is not None:
-                continue
-            if line[i - 2:i + 1] == 3 * char:
+
+                # Are we looking for a different triple quote?
+                if self.triple is not None:
+                    continue
+
+                # Triple quote starting
                 self.triple = char
                 self.triple_start = i
                 continue
+
+            # Inside a multiline quote
+            if self.triple is not None:
+                continue
+
             self.single = char
 
         # Line ended
diff --git a/setup.py b/setup.py
index ec8f8eb3..28b6969a 100644
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,7 @@ setup(
     entry_points={'console_scripts': ['jupytext = jupytext.cli:jupytext']},
     tests_require=['pytest'],
     install_requires=['nbformat>=4.0.0', 'pyyaml', 'mock;python_version<"3"'],
-    extras_require={"myst": ["myst-parser~=0.7.1; python_version >= '3.6'"]},
+    extras_require={"myst": ["myst-parser~=0.8; python_version >= '3.6'"]},
     license='MIT',
     classifiers=['Development Status :: 5 - Production/Stable',
                  'License :: OSI Approved :: MIT License',


Test patch:
diff --git a/tests/test_combine.py b/tests/test_combine.py
index d6b27a5e..0c801993 100644
--- a/tests/test_combine.py
+++ b/tests/test_combine.py
@@ -1,6 +1,10 @@
+import pytest
+from copy import deepcopy
 from nbformat.v4.nbbase import new_code_cell, new_markdown_cell, new_notebook
 from jupytext.combine import combine_inputs_with_outputs
+from jupytext.compare import compare_notebooks
 import jupytext
+from .utils import list_notebooks
 
 
 def test_combine():
@@ -114,3 +118,86 @@ def test_read_text_and_combine_with_outputs(tmpdir):
     assert nb.cells[2]['outputs']
 
     assert len(nb.cells) == 3
+
+
+@pytest.mark.parametrize('nb_file', list_notebooks('ipynb_all'))
+def test_combine_stable(nb_file):
+    nb_org = jupytext.read(nb_file)
+    nb_source = deepcopy(nb_org)
+    nb_outputs = deepcopy(nb_org)
+
+    for cell in nb_source.cells:
+        try:
+            cell.outputs = []
+        except KeyError:
+            continue
+
+    combine_inputs_with_outputs(nb_source, nb_outputs)
+    compare_notebooks(nb_source, nb_org)
+
+
+def test_combine_reorder():
+    nb_source = new_notebook(
+        cells=[new_markdown_cell('Markdown text'),
+               new_code_cell('1+1'),
+               new_code_cell('2+2'),
+               new_code_cell('3+3'),
+               new_markdown_cell('Markdown text'),
+               new_code_cell('4+4')])
+
+    nb_outputs = new_notebook(
+        cells=[new_markdown_cell('Markdown text'),
+               new_code_cell('2+2'),
+               new_code_cell('4+4'),
+               new_code_cell('1+1'),
+               new_code_cell('3+3'),
+               new_markdown_cell('Markdown text')])
+
+    nb_outputs.cells[1].outputs = ['4']
+    nb_outputs.cells[2].outputs = ['8']
+    nb_outputs.cells[3].outputs = ['2']
+    nb_outputs.cells[4].outputs = ['6']
+
+    combine_inputs_with_outputs(nb_source, nb_outputs)
+
+    assert nb_source.cells[1].outputs == ['2']
+    assert nb_source.cells[2].outputs == ['4']
+    assert nb_source.cells[3].outputs == ['6']
+    assert nb_source.cells[5].outputs == ['8']
+
+
+def test_combine_split():
+    nb_source = new_notebook(
+        cells=[new_code_cell('1+1'),
+               new_code_cell('2+2')])
+
+    nb_outputs = new_notebook(
+        cells=[new_code_cell('1+1\n2+2')])
+
+    nb_outputs.cells[0].outputs = ['4']
+
+    combine_inputs_with_outputs(nb_source, nb_outputs)
+
+    assert nb_source.cells[0].outputs == []
+    assert nb_source.cells[1].outputs == ['4']
+
+
+def test_combine_refactor():
+    nb_source = new_notebook(
+        cells=[new_code_cell('a=1'),
+               new_code_cell('a+1'),
+               new_code_cell('a+2')])
+
+    nb_outputs = new_notebook(
+        cells=[new_code_cell('b=1'),
+               new_code_cell('b+1'),
+               new_code_cell('b+2')])
+
+    nb_outputs.cells[1].outputs = ['2']
+    nb_outputs.cells[2].outputs = ['3']
+
+    combine_inputs_with_outputs(nb_source, nb_outputs)
+
+    assert nb_source.cells[0].outputs == []
+    assert nb_source.cells[1].outputs == ['2']
+    assert nb_source.cells[2].outputs == ['3']
diff --git a/tests/test_ipynb_to_myst.py b/tests/test_ipynb_to_myst.py
index bb4c6691..d5254be8 100644
--- a/tests/test_ipynb_to_myst.py
+++ b/tests/test_ipynb_to_myst.py
@@ -4,9 +4,7 @@ except ImportError:
     import mock
 from textwrap import dedent
 import pytest
-from nbformat import from_dict
 
-from jupytext.compare import compare_notebooks
 from jupytext.formats import get_format_implementation, JupytextFormatError
 from jupytext.myst import (
     myst_to_notebook,
@@ -40,7 +38,7 @@ def test_bad_code_metadata():
         myst_to_notebook(
             dedent(
                 """\
-            ```{{{0}}}
+            ```{0}
             ---
             {{a
             ---
@@ -105,6 +103,16 @@ def test_matches_mystnb():
         """
     )
     assert matches_mystnb(text) is True
+    text = dedent(
+        """\
+        ---
+        a: 1
+        ---
+        > ```{code-cell}
+          ```
+        """
+    )
+    assert matches_mystnb(text) is True
 
 
 def test_not_installed():
@@ -114,8 +122,8 @@ def test_not_installed():
 
 
 @requires_myst
-def test_store_line_numbers():
-    notebook = matches_mystnb(
+def test_add_source_map():
+    notebook = myst_to_notebook(
         dedent(
             """\
             ---
@@ -124,7 +132,7 @@ def test_store_line_numbers():
             abc
             +++
             def
-            ```{{{0}}}
+            ```{0}
             ---
             b: 2
             ---
@@ -133,37 +141,6 @@ def test_store_line_numbers():
             xyz
             """
         ).format(CODE_DIRECTIVE),
-        store_line_numbers=True,
-        return_nb=True
+        add_source_map=True,
     )
-    expected = {
-        "nbformat": 4,
-        "nbformat_minor": 4,
-        "metadata": {"a": 1},
-        "cells": [
-            {
-                "cell_type": "markdown",
-                "source": "abc",
-                "metadata": {"_source_lines": [3, 4]},
-            },
-            {
-                "cell_type": "markdown",
-                "source": "def",
-                "metadata": {"_source_lines": [5, 6]},
-            },
-            {
-                "cell_type": "code",
-                "metadata": {"b": 2, "_source_lines": [7, 12]},
-                "execution_count": None,
-                "source": "c = 3",
-                "outputs": [],
-            },
-            {
-                "cell_type": "markdown",
-                "source": "xyz",
-                "metadata": {"_source_lines": [12, 13]},
-            },
-        ],
-    }
-    notebook.nbformat_minor = 4
-    compare_notebooks(notebook, from_dict(expected))
+    assert notebook.metadata.source_map == [3, 5, 7, 12]
diff --git a/tests/test_stringparser.py b/tests/test_stringparser.py
index 36307dc1..9ed0f13e 100644
--- a/tests/test_stringparser.py
+++ b/tests/test_stringparser.py
@@ -53,3 +53,19 @@ comment that starts with four quotes
         sp.read_line(line)
 
     assert quoted == [1, 2]
+
+
+def test_long_string_ends_with_four_quotes(text="""'''This is a multiline
+comment that ends with four quotes
+''''
+
+1 + 1
+"""):
+    quoted = []
+    sp = StringParser('python')
+    for i, line in enumerate(text.splitlines()):
+        if sp.is_quoted():
+            quoted.append(i)
+        sp.read_line(line)
+
+    assert quoted == [1, 2]