Repository: mwouts/jupytext

Issue:
Hide metadata header in markdown
Is it possible, or would it be possible to hide the metedata header added when reading and writing to markdown?

For example, this is the header on one markdown document:

```markdown
---
jupyter:
  jupytext:
    text_representation:
      extension: .md
      format_name: markdown
      format_version: '1.2'
      jupytext_version: 1.4.2
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---
```

Instead, it would be great if this could be put in an inline comment so that it isn't visible when the document is rendered outside of jupyter.

(P.s. thank you very much for your work on this, it is a really excellent plugin you've built here 😄)

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/581?src=pr&el=h1) Report
> Merging [#581](https://codecov.io/gh/mwouts/jupytext/pull/581?src=pr&el=desc) into [master](https://codecov.io/gh/mwouts/jupytext/commit/a8992b01db6d9c0133d8d3dc3175c597e8028f51&el=desc) will **decrease** coverage by `12.42%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/581/graphs/tree.svg?width=650&height=150&src=pr&token=a1pNjsiILi)](https://codecov.io/gh/mwouts/jupytext/pull/581?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master     #581       +/-   ##
===========================================
- Coverage   98.98%   86.56%   -12.43%     
===========================================
  Files          89       89               
  Lines        8794     8658      -136     
===========================================
- Hits         8705     7495     -1210     
- Misses         89     1163     +1074     
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/581?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jupytext/formats.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-anVweXRleHQvZm9ybWF0cy5weQ==) | `95.65% <ø> (-1.25%)` | :arrow_down: |
| [jupytext/cli.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-anVweXRleHQvY2xpLnB5) | `95.13% <100.00%> (-1.34%)` | :arrow_down: |
| [jupytext/config.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-anVweXRleHQvY29uZmlnLnB5) | `96.24% <100.00%> (+0.08%)` | :arrow_up: |
| [jupytext/header.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-anVweXRleHQvaGVhZGVyLnB5) | `99.31% <100.00%> (+0.07%)` | :arrow_up: |
| [tests/test\_cli\_config.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jbGlfY29uZmlnLnB5) | `100.00% <100.00%> (ø)` | |
| [tests/test\_header.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9oZWFkZXIucHk=) | `100.00% <100.00%> (ø)` | |
| [tests/test\_read\_simple\_markdown.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9yZWFkX3NpbXBsZV9tYXJrZG93bi5weQ==) | `100.00% <100.00%> (ø)` | |
| [jupytext/myst.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-anVweXRleHQvbXlzdC5weQ==) | `19.89% <0.00%> (-77.42%)` | :arrow_down: |
| [tests/test\_hide\_remove\_input\_outputs\_rmarkdown.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9oaWRlX3JlbW92ZV9pbnB1dF9vdXRwdXRzX3JtYXJrZG93bi5weQ==) | `28.57% <0.00%> (-71.43%)` | :arrow_down: |
| [tests/test\_black.py](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9ibGFjay5weQ==) | `31.53% <0.00%> (-68.47%)` | :arrow_down: |
| ... and [45 more](https://codecov.io/gh/mwouts/jupytext/pull/581/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/581?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/581?src=pr&el=footer). Last update [a8992b0...997b17c](https://codecov.io/gh/mwouts/jupytext/pull/581?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eb0fa98..66fba58 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,10 +3,12 @@
 
 **Added**
 - Activated GitHub code scanning alerts
+- New option `hide_notebook_metadata` to encapsulate the notebook metadata in an HTML comment (#527)
 
 **Changed**
 - Install Jupytext from source on MyBinder to avoid cache issues (#567)
 - Skip the tests that execute a notebook on Windows to avoid timeout issues (#489)
+- The `# %%` cell marker has the same indentation as the first line in the cell (#562)
 
 **Fixed**
 - Configured coverage targets in `codecov.yml`
diff --git a/docs/config.md b/docs/config.md
index 9745d3f..002b44e 100644
--- a/docs/config.md
+++ b/docs/config.md
@@ -105,6 +105,8 @@ It is possible to filter nested metadata. For example, if you want to preserve t
 default_notebook_metadata_filter = "-jupytext.text_representation.jupytext_version"
 ```
 
+Finally, to hide the notebook metadata in an HTML comment in Markdown files, use the option `hide_notebook_metadata`.
+
 ### More options
 
 There are a couple more options available - please have a look at the `JupytextConfiguration` class in [config.py](https://github.com/mwouts/jupytext/blob/master/jupytext/config.py).
diff --git a/jupytext/cell_reader.py b/jupytext/cell_reader.py
index b3d3de7..627d58b 100644
--- a/jupytext/cell_reader.py
+++ b/jupytext/cell_reader.py
@@ -747,9 +747,9 @@ class DoublePercentScriptCellReader(LightScriptCellReader):
         script = _SCRIPT_EXTENSIONS[self.ext]
         self.default_language = default_language or script["language"]
         self.comment = script["comment"]
-        self.start_code_re = re.compile(r"^{}\s*%%(%*)\s(.*)$".format(self.comment))
+        self.start_code_re = re.compile(r"^\s*{}\s*%%(%*)\s(.*)$".format(self.comment))
         self.alternative_start_code_re = re.compile(
-            r"^{}\s*(%%|<codecell>|In\[[0-9 ]*\]:?)\s*$".format(self.comment)
+            r"^\s*{}\s*(%%|<codecell>|In\[[0-9 ]*\]:?)\s*$".format(self.comment)
         )
         self.explicit_soc = True
 
diff --git a/jupytext/cell_to_text.py b/jupytext/cell_to_text.py
index 369036c..b9a4f9a 100644
--- a/jupytext/cell_to_text.py
+++ b/jupytext/cell_to_text.py
@@ -457,10 +457,18 @@ class DoublePercentCellExporter(BaseCellExporter):  # pylint: disable=W0223
         options = metadata_to_double_percent_options(
             self.metadata, self.cell_metadata_json
         )
+        indent = ""
+        if self.is_code() and active and self.source:
+            first_line = self.source[0]
+            if first_line.strip():
+                left_space = re.compile(r"^(\s*)").match(first_line)
+                if left_space:
+                    indent = left_space.groups()[0]
+
         if options.startswith("%") or not options:
-            lines = [self.comment + " %%" + options]
+            lines = [indent + self.comment + " %%" + options]
         else:
-            lines = [self.comment + " %% " + options]
+            lines = [indent + self.comment + " %% " + options]
 
         if self.is_code() and active:
             source = copy(self.source)
diff --git a/jupytext/cli.py b/jupytext/cli.py
index 66e208f..5c8b7c5 100644
--- a/jupytext/cli.py
+++ b/jupytext/cli.py
@@ -409,9 +409,14 @@ def jupytext_single_file(nb_file, args, log):
 
     # I. ### Read the notebook ###
     fmt = copy(args.input_format) or {}
-    if config:
-        config.set_default_format_options(fmt)
-    set_format_options(fmt, args.format_options)
+    if not fmt:
+        ext = os.path.splitext(nb_file)[1]
+        if ext:
+            fmt = {"extension": ext}
+    if fmt:
+        if config:
+            config.set_default_format_options(fmt)
+        set_format_options(fmt, args.format_options)
     log(
         "[jupytext] Reading {}{}".format(
             nb_file if nb_file != "-" else "stdin",
@@ -422,19 +427,12 @@ def jupytext_single_file(nb_file, args, log):
     )
 
     notebook = read(nb_file, fmt=fmt)
-    if not fmt:
+    if "extension" in fmt and "format_name" not in fmt:
         text_representation = notebook.metadata.get("jupytext", {}).get(
             "text_representation", {}
         )
-        ext = os.path.splitext(nb_file)[1]
-        if text_representation.get("extension") == ext:
-            fmt = {
-                key: text_representation[key]
-                for key in text_representation
-                if key in ["extension", "format_name"]
-            }
-        elif ext:
-            fmt = {"extension": ext}
+        if text_representation.get("extension") == fmt["extension"]:
+            fmt["format_name"] = text_representation["format_name"]
 
     if config and "formats" not in notebook.metadata.get("jupytext", {}):
         default_formats = config.default_formats(nb_file)
diff --git a/jupytext/config.py b/jupytext/config.py
index 4ecd69f..7f005de 100644
--- a/jupytext/config.py
+++ b/jupytext/config.py
@@ -62,14 +62,21 @@ class JupytextConfiguration(Configurable):
 
     default_notebook_metadata_filter = Unicode(
         u"",
-        help="Cell metadata that should be save in the text representations. "
+        help="Notebook metadata that should be save in the text representations. "
         "Examples: 'all', '-all', 'widgets,nteract', 'kernelspec,jupytext-all'",
         config=True,
     )
 
+    hide_notebook_metadata = Enum(
+        values=[True, False],
+        allow_none=True,
+        help="Should the notebook metadata be wrapped into an HTML comment in the Markdown format?",
+        config=True,
+    )
+
     default_cell_metadata_filter = Unicode(
         u"",
-        help="Notebook metadata that should be saved in the text representations. "
+        help="Cell metadata that should be saved in the text representations. "
         "Examples: 'all', 'hide_input,hide_output'",
         config=True,
     )
@@ -124,6 +131,10 @@ class JupytextConfiguration(Configurable):
             format_options.setdefault(
                 "cell_metadata_filter", self.default_cell_metadata_filter
             )
+        if self.hide_notebook_metadata is not None:
+            format_options.setdefault(
+                "hide_notebook_metadata", self.hide_notebook_metadata
+            )
         if self.comment_magics is not None:
             format_options.setdefault("comment_magics", self.comment_magics)
         if self.split_at_heading:
diff --git a/jupytext/formats.py b/jupytext/formats.py
index 026092d..0bada49 100644
--- a/jupytext/formats.py
+++ b/jupytext/formats.py
@@ -692,6 +692,7 @@ def short_form_multiple_formats(jupytext_formats):
 _VALID_FORMAT_INFO = ["extension", "format_name", "suffix", "prefix"]
 _BINARY_FORMAT_OPTIONS = [
     "comment_magics",
+    "hide_notebook_metadata",
     "split_at_heading",
     "rst2md",
     "cell_metadata_json",
diff --git a/jupytext/header.py b/jupytext/header.py
index ddabbbd..9151071 100644
--- a/jupytext/header.py
+++ b/jupytext/header.py
@@ -116,6 +116,12 @@ def metadata_and_cell_to_header(notebook, metadata, text_format, ext):
     if header:
         header = ["---"] + header + ["---"]
 
+        if (
+            metadata.get("jupytext", {}).get("hide_notebook_metadata", False)
+            and text_format.format_name == "markdown"
+        ):
+            header = ["<!--", ""] + header + ["", "-->"]
+
     return comment_lines(header, text_format.header_prefix), lines_to_next_cell
 
 
@@ -142,10 +148,13 @@ def header_to_metadata_and_cell(lines, header_prefix, ext=None):
 
     header = []
     jupyter = []
-    injupyter = False
+    in_jupyter = False
+    in_html_div = False
+
+    start = 0
+    started = False
     ended = False
     metadata = {}
-    start = 0
     i = -1
 
     comment = "#" if header_prefix == "#'" else header_prefix
@@ -167,27 +176,37 @@ def header_to_metadata_and_cell(lines, header_prefix, ext=None):
                 metadata.setdefault("jupytext", {})["encoding"] = line
                 start = i + 1
                 continue
-
         if not line.startswith(header_prefix):
             break
+        if not comment:
+            if line.strip().startswith("<!--"):
+                in_html_div = True
+                continue
 
-        line = uncomment_line(line, header_prefix)
-
-        if i == start:
-            if _HEADER_RE.match(line):
+        if in_html_div:
+            if ended:
+                if "-->" in line:
+                    break
+            if not started and not line.strip():
                 continue
-            break
 
-        if i > start and _HEADER_RE.match(line):
-            ended = True
-            break
+        line = uncomment_line(line, header_prefix)
+        if _HEADER_RE.match(line):
+            if not started:
+                started = True
+                continue
+            else:
+                ended = True
+                if in_html_div:
+                    continue
+                break
 
         if _JUPYTER_RE.match(line):
-            injupyter = True
+            in_jupyter = True
         elif line and not _LEFTSPACE_RE.match(line):
-            injupyter = False
+            in_jupyter = False
 
-        if injupyter:
+        if in_jupyter:
             jupyter.append(line)
         else:
             header.append(line)
diff --git a/jupytext/version.py b/jupytext/version.py
index b6d1df7..5580906 100644
--- a/jupytext/version.py
+++ b/jupytext/version.py
@@ -1,3 +1,3 @@
 """Jupytext's version number"""
 
-__version__ = "1.5.2"
+__version__ = "1.6.0-dev"


Test patch:
diff --git a/tests/test_cli_config.py b/tests/test_cli_config.py
index decb3a0..c83fea7 100644
--- a/tests/test_cli_config.py
+++ b/tests/test_cli_config.py
@@ -3,6 +3,7 @@ from nbformat.v4.nbbase import new_notebook, new_code_cell
 from jupytext.cli import jupytext
 from jupytext.jupytext import read, write
 from jupytext.header import header_to_metadata_and_cell
+from jupytext.compare import compare
 
 
 def test_default_jupytext_formats(tmpdir):
@@ -89,3 +90,38 @@ def test_save_using_preferred_and_default_format(config, tmpdir):
     # read py file
     nb_py = read(str(tmp_py))
     assert nb_py.metadata["jupytext"]["text_representation"]["format_name"] == "percent"
+
+
+def test_hide_notebook_metadata(tmpdir, no_jupytext_version_number):
+    tmpdir.join(".jupytext").write("hide_notebook_metadata = true")
+    tmp_ipynb = tmpdir.join("notebook.ipynb")
+    tmp_md = tmpdir.join("notebook.md")
+
+    nb = new_notebook(
+        cells=[new_code_cell("1 + 1")], metadata={"jupytext": {"formats": "ipynb,md"}}
+    )
+
+    write(nb, str(tmp_ipynb))
+    jupytext([str(tmp_ipynb), "--sync"])
+
+    with open(str(tmp_md)) as stream:
+        text_md = stream.read()
+
+    compare(
+        text_md,
+        """<!--
+
+---
+jupyter:
+  jupytext:
+    formats: ipynb,md
+    hide_notebook_metadata: true
+---
+
+-->
+
+```python
+1 + 1
+```
+""",
+    )
diff --git a/tests/test_header.py b/tests/test_header.py
index 9801e0b..8977ba0 100644
--- a/tests/test_header.py
+++ b/tests/test_header.py
@@ -148,3 +148,41 @@ jupyter:
     compare(actual, markdown)
     nb2 = jupytext.reads(markdown, ".md")
     compare(nb2, notebook)
+
+
+def test_header_in_html_comment():
+    text = """<!--
+
+---
+jupyter:
+  title: Sample header
+---
+
+-->
+"""
+    lines = text.splitlines()
+    metadata, _, cell, _ = header_to_metadata_and_cell(lines, "")
+
+    assert metadata == {"title": "Sample header"}
+    assert cell is None
+
+
+def test_header_to_html_comment(no_jupytext_version_number):
+    metadata = {"jupytext": {"mainlanguage": "python", "hide_notebook_metadata": True}}
+    nb = new_notebook(metadata=metadata, cells=[])
+    header, lines_to_next_cell = metadata_and_cell_to_header(
+        nb, metadata, get_format_implementation(".md"), ".md"
+    )
+    compare(
+        "\n".join(header),
+        """<!--
+
+---
+jupyter:
+  jupytext:
+    hide_notebook_metadata: true
+    mainlanguage: python
+---
+
+-->""",
+    )
diff --git a/tests/test_read_simple_markdown.py b/tests/test_read_simple_markdown.py
index ad09669..728713f 100644
--- a/tests/test_read_simple_markdown.py
+++ b/tests/test_read_simple_markdown.py
@@ -916,3 +916,37 @@ jupyter:
     compare(md2, md)
     nb2 = jupytext.reads(md, "md")
     compare_notebooks(nb2, nb)
+
+
+def test_hide_notebook_metadata(
+    no_jupytext_version_number,
+    nb=new_notebook(
+        metadata={
+            "jupytext": {"hide_notebook_metadata": True},
+            "kernelspec": {
+                "display_name": "Python 3",
+                "language": "python",
+                "name": "python3",
+            },
+        }
+    ),
+    md="""<!--
+
+---
+jupyter:
+  jupytext:
+    hide_notebook_metadata: true
+  kernelspec:
+    display_name: Python 3
+    language: python
+    name: python3
+---
+
+-->
+""",
+):
+    """Test the hide_notebook_metadata option"""
+    md2 = jupytext.writes(nb, "md")
+    compare(md2, md)
+    nb2 = jupytext.reads(md, "md")
+    compare_notebooks(nb2, nb)
diff --git a/tests/test_read_simple_percent.py b/tests/test_read_simple_percent.py
index 60839e8..bfc79ff 100644
--- a/tests/test_read_simple_percent.py
+++ b/tests/test_read_simple_percent.py
@@ -502,3 +502,32 @@ def test_docstring_with_quadruple_quote(
     py = jupytext.writes(nb, "py:percent")
     nb2 = jupytext.reads(py, "py")
     compare_notebooks(nb2, nb)
+
+
+def test_cell_marker_has_same_indentation_as_code(
+    text="""# %%
+if __name__ == '__main__':
+    print(1)
+
+    # %%
+    # INDENTED COMMENT
+    print(2)
+""",
+    nb_expected=new_notebook(
+        cells=[
+            new_code_cell(
+                """if __name__ == '__main__':
+    print(1)"""
+            ),
+            new_code_cell(
+                """    # INDENTED COMMENT
+    print(2)"""
+            ),
+        ]
+    ),
+):
+    """The cell marker should have the same indentation as the first code line. See issue #562"""
+    nb_actual = jupytext.reads(text, fmt="py:percent")
+    compare_notebooks(nb_actual, nb_expected)
+    text_actual = jupytext.writes(nb_actual, fmt="py:percent")
+    compare(text_actual, text)