Repository: mwouts/jupytext
Pull request #601 (2020-08-30 10:30:09)

Jupytext depends on markdownitpy
This is an alternative to #599 which has the dependency on `markdown-it-py` only when Python >= 3.6.

codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/601?src=pr&el=h1) Report
> Merging [#601](https://codecov.io/gh/mwouts/jupytext/pull/601?src=pr&el=desc) into [master](https://codecov.io/gh/mwouts/jupytext/commit/b192d5bfa08b24d7dca728779adb6c1fcb4fb0fd?el=desc) will **decrease** coverage by `0.20%`.
> The diff coverage is `91.66%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/601/graphs/tree.svg?width=650&height=150&src=pr&token=a1pNjsiILi)](https://codecov.io/gh/mwouts/jupytext/pull/601?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #601      +/-   ##
==========================================
- Coverage   99.00%   98.80%   -0.21%     
==========================================
  Files          90       90              
  Lines        8951     9034      +83     
==========================================
+ Hits         8862     8926      +64     
- Misses         89      108      +19     
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/601?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jupytext/myst.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-anVweXRleHQvbXlzdC5weQ==) | `96.71% <91.48%> (-0.70%)` | :arrow_down: |
| [tests/utils.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-dGVzdHMvdXRpbHMucHk=) | `100.00% <100.00%> (ø)` | |
| [jupytext/reraise.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-anVweXRleHQvcmVyYWlzZS5weQ==) | `25.00% <0.00%> (-50.00%)` | :arrow_down: |
| [tests/test\_isort.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9pc29ydC5weQ==) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [tests/test\_read\_simple\_pandoc.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9yZWFkX3NpbXBsZV9wYW5kb2MucHk=) | `89.47% <0.00%> (-10.53%)` | :arrow_down: |
| [jupytext/pandoc.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-anVweXRleHQvcGFuZG9jLnB5) | `87.93% <0.00%> (-8.63%)` | :arrow_down: |
| [tests/test\_cm\_config.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jbV9jb25maWcucHk=) | `100.00% <0.00%> (ø)` | |
| [tests/test\_cli\_config.py](https://codecov.io/gh/mwouts/jupytext/pull/601/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jbGlfY29uZmlnLnB5) | `100.00% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/601?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/601?src=pr&el=footer). Last update [b192d5b...0e2f1c7](https://codecov.io/gh/mwouts/jupytext/pull/601?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

chrisjsewell: Yep that looks fine thanks. Two minor things:

As you see in the CI, if you make a PR to master from an "internal" branch you get duplication. To remove this we use:

```
on:
  push:
    branches: [master]
    tags:
      - 'v*'
  pull_request:
```

I made #602, to also update the import error message and documentation
chrisjsewell: Another FYI:
I have recently found using [tox](https://tox.readthedocs.io) to be really helpful for not having to manually create/maintain development environment (given that I'm working on over 10 different repos!)

Here's the `tox.ini` I'm using for jupytext:

```ini
# To use tox, see https://tox.readthedocs.io
# Simply pip or conda install tox
# If you use conda, you may also want to install tox-conda
# then run `tox` or `tox -- {pytest args}`
# To run in parallel using `tox -p` (this does not appear to work for this repo)

# To rebuild the tox environment, for example when dependencies change, use
# `tox -r`

# Note: if the following error is encountered: `ImportError while loading conftest`
# then then deleting compiled files has been found to fix it: `find . -name \*.pyc -delete`

[tox]
envlist = py{35,36,37,38}

[testenv:py{35,36,37,38}]
; recreate = false
deps = -rrequirements-dev.txt
commands = pytest {posargs}
```
mwouts: Thank you @chrisjsewell ! This is very helpful. I'll integrate this PR, as well as your update on the documentation and warnings, and open a follow-up issue to keep track of your multiple suggestions :smile: .

Patch:
diff --git a/.github/workflows/continuous-integration-conda.yml b/.github/workflows/continuous-integration-conda.yml
index d3303dac..14e33af8 100644
--- a/.github/workflows/continuous-integration-conda.yml
+++ b/.github/workflows/continuous-integration-conda.yml
@@ -65,7 +65,7 @@ jobs:
           # install sphinx_gallery and matplotlib if available
           conda install sphinx-gallery --freeze-installed
           # myst-parser
-          conda install 'myst-parser>=0.8' 'myst-parser<0.9' --freeze-installed
+          conda install 'markdown-it-py>=0.5' 'markdown-it-py<0.6' --freeze-installed
           exit 0
       - name: Conda list
         shell: pwsh
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8f91f511..8a5ddc51 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,7 @@
 - The `# %%` cell marker has the same indentation as the first line in the cell (#562)
 - The `md:myst` and `md:pandoc` are always included in the Jupytext formats, and an informative runtime
 error will occur if the required dependencies, resp. `myst-parser` and `pandoc`, are not installed. (#556)
+- Jupytext now depends on `markdown-it-py` and always feature the MyST-Markdown format (Python 3.6 and above, #591)  
 
 **Fixed**
 - Configured coverage targets in `codecov.yml`
diff --git a/jupytext/cell_reader.py b/jupytext/cell_reader.py
index 627d58b2..551a1fa9 100644
--- a/jupytext/cell_reader.py
+++ b/jupytext/cell_reader.py
@@ -4,6 +4,7 @@ import re
 from copy import copy
 from nbformat.v4.nbbase import new_code_cell, new_raw_cell, new_markdown_cell
 from .languages import _SCRIPT_EXTENSIONS
+from .doxygen import doxygen_to_markdown
 
 # Sphinx Gallery is an optional dependency. And we intercept the SyntaxError for #301
 try:
@@ -122,6 +123,7 @@ class BaseCellReader(object):
         self.cell_type = None
         self.language = None
         self.cell_metadata_json = fmt.get("cell_metadata_json", False)
+        self.doxygen_equation_markers = fmt.get("doxygen_equation_markers", False)
 
     def read(self, lines):
         """Read one cell from the given lines, and return the cell,
@@ -457,6 +459,8 @@ class MarkdownCellReader(BaseCellReader):
     def uncomment_code_and_magics(self, lines):
         if self.cell_type == "code" and self.comment_magics:
             lines = uncomment_magic(lines, self.language)
+        if self.cell_type == "markdown" and self.doxygen_equation_markers:
+            lines = doxygen_to_markdown("\n".join(lines)).splitlines()
         return lines
 
 
diff --git a/jupytext/cell_to_text.py b/jupytext/cell_to_text.py
index b9a4f9ac..9a241221 100644
--- a/jupytext/cell_to_text.py
+++ b/jupytext/cell_to_text.py
@@ -15,6 +15,7 @@ from .magics import comment_magic, escape_code_start, need_explicit_marker
 from .cell_reader import LightScriptCellReader, MarkdownCellReader, RMarkdownCellReader
 from .languages import _SCRIPT_EXTENSIONS
 from .pep8 import pep8_lines_between_cells
+from .doxygen import markdown_to_doxygen
 
 
 def cell_source(cell):
@@ -61,6 +62,7 @@ class BaseCellExporter(object):
         )
         self.cell_metadata_json = self.fmt.get("cell_metadata_json", False)
         self.use_runtools = self.fmt.get("use_runtools", False)
+        self.doxygen_equation_markers = self.fmt.get("doxygen_equation_markers", False)
 
         # how many blank lines before next cell
         self.lines_to_next_cell = cell.metadata.get("lines_to_next_cell")
@@ -185,6 +187,9 @@ class MarkdownCellExporter(BaseCellExporter):
     def cell_to_text(self):
         """Return the text representation of a cell"""
         if self.cell_type == "markdown":
+            if self.doxygen_equation_markers and self.cell_type == "markdown":
+                self.source = markdown_to_doxygen("\n".join(self.source)).splitlines()
+
             # Is an explicit region required?
             if self.metadata:
                 protect = True
diff --git a/jupytext/cli.py b/jupytext/cli.py
index 3312d5e0..3b988676 100644
--- a/jupytext/cli.py
+++ b/jupytext/cli.py
@@ -459,6 +459,8 @@ def jupytext_single_file(nb_file, args, log):
 
     # Compute actual extension when using script/auto, and update nb_dest if necessary
     dest_fmt = args.output_format
+    if dest_fmt and config:
+        config.set_default_format_options(dest_fmt)
     if dest_fmt and dest_fmt["extension"] == ".auto":
         dest_fmt = check_auto_ext(dest_fmt, notebook.metadata, "--to")
         if not args.output and nb_file != "-":
diff --git a/jupytext/config.py b/jupytext/config.py
index 7f005de4..b298f11c 100644
--- a/jupytext/config.py
+++ b/jupytext/config.py
@@ -100,6 +100,13 @@ class JupytextConfiguration(Configurable):
         config=True,
     )
 
+    doxygen_equation_markers = Bool(
+        False,
+        help="Should equation markers use the DOxygen format? "
+        "(see https://github.com/mwouts/jupytext/issues/517)",
+        config=True,
+    )
+
     outdated_text_notebook_margin = Float(
         1.0,
         help="Refuse to overwrite inputs of a ipynb notebooks with those of a "
@@ -139,6 +146,10 @@ class JupytextConfiguration(Configurable):
             format_options.setdefault("comment_magics", self.comment_magics)
         if self.split_at_heading:
             format_options.setdefault("split_at_heading", self.split_at_heading)
+        if self.doxygen_equation_markers:
+            format_options.setdefault(
+                "doxygen_equation_markers", self.doxygen_equation_markers
+            )
         if not read and self.default_cell_markers:
             format_options.setdefault("cell_markers", self.default_cell_markers)
         if read and self.sphinx_convert_rst2md:
diff --git a/jupytext/doxygen.py b/jupytext/doxygen.py
new file mode 100644
index 00000000..17eaa169
--- /dev/null
+++ b/jupytext/doxygen.py
@@ -0,0 +1,20 @@
+"""Convert Markdown equations to doxygen equations and back
+See https://github.com/mwouts/jupytext/issues/517"""
+
+import re
+
+
+def markdown_to_doxygen(string):
+    """Markdown to Doxygen equations"""
+    long_equations = re.sub(
+        r"(?<!\\)\$\$(.*?)(?<!\\)\$\$", r"\\f[\g<1>\\f]", string, flags=re.DOTALL
+    )
+    inline_equations = re.sub(r"(?<!(\\|\$))\$(?!\$)", r"\\f$", long_equations)
+    return inline_equations
+
+
+def doxygen_to_markdown(string):
+    """Doxygen to Markdown equations"""
+    long_equations = re.sub(r"\\f\[(.*?)\\f\]", r"$$\g<1>$$", string, flags=re.DOTALL)
+    inline_equations = re.sub(r"\\f\$", "$", long_equations)
+    return inline_equations
diff --git a/jupytext/formats.py b/jupytext/formats.py
index 10731eaa..89691ae8 100644
--- a/jupytext/formats.py
+++ b/jupytext/formats.py
@@ -702,6 +702,7 @@ _BINARY_FORMAT_OPTIONS = [
     "rst2md",
     "cell_metadata_json",
     "use_runtools",
+    "doxygen_equation_markers",
 ]
 _VALID_FORMAT_OPTIONS = _BINARY_FORMAT_OPTIONS + [
     "notebook_metadata_filter",
diff --git a/jupytext/myst.py b/jupytext/myst.py
index 72d565f7..baa28038 100644
--- a/jupytext/myst.py
+++ b/jupytext/myst.py
@@ -4,19 +4,19 @@ myst formatted text documents and notebooks.
 """
 import json
 import warnings
+import re
+from textwrap import dedent
 
 import nbformat as nbf
 import yaml
-from .reraise import reraise
 
 try:
-    import myst_parser
-    from myst_parser.main import default_parser
-    from myst_parser.parse_directives import DirectiveParsingError, parse_directive_text
-except ImportError as err:
-    myst_parser = None
-    DirectiveParsingError = Exception
-    default_parser = parse_directive_text = reraise(err)
+    from markdown_it import MarkdownIt
+    from markdown_it.extensions.front_matter import front_matter_plugin
+    from markdown_it.extensions.myst_blocks import myst_block_plugin
+    from markdown_it.extensions.myst_role import myst_role_plugin
+except ImportError:
+    MarkdownIt = None
 
 MYST_FORMAT_NAME = "myst"
 CODE_DIRECTIVE = "{code-cell}"
@@ -24,14 +24,8 @@ RAW_DIRECTIVE = "{raw-cell}"
 
 
 def is_myst_available():
-    """Whether the myst-parser package is available."""
-    if myst_parser is None:
-        return False
-    major, minor = myst_parser.__version__.split(".")[:2]
-    if int(major) < 1 and int(minor) < 8:
-        warnings.warn("The installed myst-parser version is less than the required 0.8")
-        return False
-    return True
+    """Whether the markdown-it-py package is available."""
+    return MarkdownIt is not None
 
 
 def raise_if_myst_is_not_available():
@@ -43,10 +37,8 @@ def raise_if_myst_is_not_available():
 
 
 def myst_version():
-    """The major version of myst parser."""
-    if is_myst_available():
-        return ".".join(myst_parser.__version__.split(".")[:2])
-    return "N/A"
+    """The version of myst."""
+    return 0.12
 
 
 def myst_extensions(no_md=False):
@@ -56,6 +48,20 @@ def myst_extensions(no_md=False):
     return [".md", ".myst", ".mystnb", ".mnb"]
 
 
+def get_parser():
+    """Return the markdown-it parser to use."""
+    parser = (
+        MarkdownIt("commonmark")
+        .enable("table")
+        .use(front_matter_plugin)
+        .use(myst_block_plugin)
+        .use(myst_role_plugin)
+        # we only need to parse block level components (for efficiency)
+        .disable("inline", True)
+    )
+    return parser
+
+
 def matches_mystnb(
     text,
     ext=None,
@@ -79,9 +85,7 @@ def matches_mystnb(
         return False
 
     try:
-        # parse markdown file up to the block level (i.e. don't worry about inline text)
-        parser = default_parser("html", disable_syntax=["inline"])
-        tokens = parser.parse(text + "\n")
+        tokens = get_parser().parse(text + "\n")
     except (TypeError, ValueError) as err:
         warnings.warn("myst-parse failed unexpectedly: {}".format(err))
         return False
@@ -164,13 +168,6 @@ def from_nbnode(value):
     return value
 
 
-class MockDirective:
-    option_spec = {"options": True}
-    required_arguments = 0
-    optional_arguments = 1
-    has_content = True
-
-
 class MystMetadataParsingError(Exception):
     """Error when parsing metadata from myst formatted text"""
 
@@ -184,23 +181,56 @@ def strip_blank_lines(text):
 
 
 def read_fenced_cell(token, cell_index, cell_type):
-    """Return cell options and body"""
-    try:
-        _, options, body_lines = parse_directive_text(
-            directive_class=MockDirective,
-            argument_str="",
-            content=token.content,
-            validate_options=False,
-        )
-    except DirectiveParsingError as err:
-        raise MystMetadataParsingError(
-            "{0} cell {1} at line {2} could not be read: {3}".format(
-                cell_type, cell_index, token.map[0] + 1, err
-            )
-        )
+    """Parse (and validate) the full directive text."""
+    content = token.content
+    error_msg = "{0} cell {1} at line {2} could not be read: ".format(
+        cell_type, cell_index, token.map[0] + 1
+    )
+
+    body_lines, options = parse_directive_options(content, error_msg)
+
+    # remove first line of body if blank
+    # this is to allow space between the options and the content
+    if body_lines and not body_lines[0].strip():
+        body_lines = body_lines[1:]
+
     return options, body_lines
 
 
+def parse_directive_options(content, error_msg):
+    """Parse (and validate) the directive option section."""
+    options = {}
+    if content.startswith("---"):
+        content = "\n".join(content.splitlines()[1:])
+        match = re.search(r"^-{3,}", content, re.MULTILINE)
+        if match:
+            yaml_block = content[: match.start()]
+            content = content[match.end() + 1 :]
+        else:
+            yaml_block = content
+            content = ""
+        yaml_block = dedent(yaml_block)
+        try:
+            options = yaml.safe_load(yaml_block) or {}
+        except (yaml.parser.ParserError, yaml.scanner.ScannerError) as error:
+            raise MystMetadataParsingError(error_msg + "Invalid YAML; " + str(error))
+    elif content.lstrip().startswith(":"):
+        content_lines = content.splitlines()  # type: list
+        yaml_lines = []
+        while content_lines:
+            if not content_lines[0].lstrip().startswith(":"):
+                break
+            yaml_lines.append(content_lines.pop(0).lstrip()[1:])
+        yaml_block = "\n".join(yaml_lines)
+        content = "\n".join(content_lines)
+        try:
+            options = yaml.safe_load(yaml_block) or {}
+        except (yaml.parser.ParserError, yaml.scanner.ScannerError) as error:
+            raise MystMetadataParsingError(error_msg + "Invalid YAML; " + str(error))
+
+    return content.splitlines(), options
+
+
 def read_cell_metadata(token, cell_index):
     """Return cell metadata"""
     metadata = {}
@@ -244,9 +274,7 @@ def myst_to_notebook(
     """
     raise_if_myst_is_not_available()
 
-    # parse markdown file up to the block level (i.e. don't worry about inline text)
-    parser = default_parser("html", disable_syntax=["inline"])
-    tokens = parser.parse(text + "\n")
+    tokens = get_parser().parse(text + "\n")
     lines = text.splitlines()
     md_start_line = 0
 
diff --git a/requirements.txt b/requirements.txt
index 06c641ab..a520266d 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,3 +1,4 @@
+#markdown-it-py~=0.5.2 #Python>=3.6
 nbformat>=4.0.0
 pyyaml
 toml
diff --git a/setup.py b/setup.py
index 29289a1e..60f0ec54 100644
--- a/setup.py
+++ b/setup.py
@@ -53,9 +53,16 @@ setup(
     ],
     entry_points={"console_scripts": ["jupytext = jupytext.cli:jupytext"]},
     tests_require=["pytest"],
-    install_requires=["nbformat>=4.0.0", "pyyaml", "toml", 'mock;python_version<"3"'],
+    install_requires=[
+        "markdown-it-py~=0.5.2; python_version >= '3.6'",
+        "nbformat>=4.0.0",
+        "pyyaml",
+        "toml",
+        'mock; python_version<"3"',
+    ],
     extras_require={
-        "myst": ["myst-parser~=0.8.0; python_version >= '3.6'"],
+        # left for back-compatibility
+        "myst": [],
         "toml": ["toml"],
     },
     license="MIT",


Test patch:
diff --git a/tests/test_doxygen.py b/tests/test_doxygen.py
new file mode 100644
index 00000000..936c1637
--- /dev/null
+++ b/tests/test_doxygen.py
@@ -0,0 +1,72 @@
+import pytest
+import nbformat
+import jupytext
+from jupytext.cli import jupytext as jupytext_cli
+from jupytext.doxygen import doxygen_to_markdown, markdown_to_doxygen
+from jupytext.compare import compare, compare_notebooks
+
+SAMPLE_MARKDOWN = """A latex formula $$1+1$$, another one $$2+2
+$$
+
+An inline formula $3+3$
+"""
+
+SAMPLE_DOXYGEN = """A latex formula \\f[1+1\\f], another one \\f[2+2
+\\f]
+
+An inline formula \\f$3+3\\f$
+"""
+
+
+def test_markdown_to_doxygen():
+    compare(markdown_to_doxygen(SAMPLE_MARKDOWN), SAMPLE_DOXYGEN)
+
+
+def test_doxygen_to_markdown():
+    compare(doxygen_to_markdown(SAMPLE_DOXYGEN), SAMPLE_MARKDOWN)
+
+
+@pytest.mark.parametrize(
+    "latex,doxygen",
+    [
+        ("$$", "$$"),
+        ("$1+1$", "\\f$1+1\\f$"),
+        ("$1\\$ $", "\\f$1\\$ \\f$"),
+        ("$$2+2$$", "\\f[2+2\\f]"),
+        ("$$2+2$$ then $$3+3$$", "\\f[2+2\\f] then \\f[3+3\\f]"),
+        ("$$2+2\n$$", "\\f[2+2\n\\f]"),
+        ("$$1\\$+2\\$=3\\$$$", "\\f[1\\$+2\\$=3\\$\\f]"),
+    ],
+)
+def test_simple_equations_to_doxygen_and_back(latex, doxygen):
+    assert markdown_to_doxygen(latex) == doxygen
+    assert doxygen_to_markdown(doxygen) == latex
+
+
+def test_doxygen_equation_markers(tmpdir):
+    cfg_file = tmpdir.join("jupytext.toml")
+    nb_file = tmpdir.join("notebook.ipynb")
+    md_file = tmpdir.join("notebook.md")
+
+    cfg_file.write("hide_notebook_metadata = true\ndoxygen_equation_markers = true")
+    nb = jupytext.reads(SAMPLE_MARKDOWN, "md")
+    del nb.metadata["jupytext"]
+    nbformat.write(nb, str(nb_file))
+
+    jupytext_cli([str(nb_file), "--to", "md"])
+    md = md_file.read()
+
+    # Doxygen equation markers
+    assert "$$" not in md
+    assert "\\f[" in md
+    assert "\\f]" in md
+    assert "\\f$" in md
+    assert "\\f$" in md
+
+    # Metadata hidden
+    assert md.startswith("<!--\n\n---\n")
+
+    jupytext_cli([str(md_file), "--to", "notebook"])
+    nb2 = nbformat.read(str(nb_file), as_version=4)
+
+    compare_notebooks(nb2, nb)