Repository: mwouts/jupytext

Issue:
Set-formats on fresh text file based on configuration
This is an extension of a related issue #485. 

I would like to be able to set the format on a fresh text file or modify the text file based on the current configuration (local if available else global) without specifying the argument to `--set-formats`.

For example, for the configuration in local `.jupyter`
```
# Pair notebooks
default_jupytext_formats = "markdown//md,notebooks//ipynb,scripts//py:percent"
```
and we have a a directory structure with a `.md` file in `markdown` directory:
```
.
├── .jupytext
├── markdown
│   └── intro.md
├── notebooks
└── scripts
```
And the pairing setup of the markdown file could be:
* Within parent directory
```
jupytext --set-formats markdown/00-intro.md
```
* or within `markdown` directory
```
jupytext --set-formats 00-intro.md
```
Above two command examples should be informed by the configuration that the markdown file is in the `markdown` directory.

To check what the current version (`1.5.0`) does, I ran both commands and got the same error
```
Traceback (most recent call last):
  File "C:\Users\at17\.conda\envs\sthm\Scripts\jupytext-script.py", line 9, in <module>
    sys.exit(jupytext())
  File "C:\Users\at17\.conda\envs\sthm\lib\site-packages\jupytext\cli.py", line 376, in jupytext
    exit_code += jupytext_single_file(nb_file, args, log)
  File "C:\Users\at17\.conda\envs\sthm\lib\site-packages\jupytext\cli.py", line 389, in jupytext_single_file
    raise ValueError("Cannot sync a notebook on stdin")
ValueError: Cannot sync a notebook on stdin
```
Is this usecase not yet supported or is there a different command I should use?
Lastly, I wonder if commandline operations can be used with the configuration which will save me from typing same arguments repetitively. 

Discussion:

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5cd3a5d..e32faed 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,8 +7,10 @@
 - New option `root_level_metadata_as_raw_cell`. Set it to `False` if you don't want to see root level metadata
 of R Markdown notebooks as a raw cell in Jupyter (#415)
 - New option `doxygen_equation_markers` to translate Markdown equations into Doxygen equations (#517)
+- New option `custom_cell_magics` to comment out cells starting with user-specific cell magics (#513)
 - Tested `isort` on notebooks (#553)
 - `jupytext notebook.ipynb --to filename.py` will warn that `--to` is used in place of `--output`.
+- `jupytext --set-formats filename.py` will suggest to use `--sync` instead of `--set-formats` (#544)
 - Warn if 'Include Metadata' is off when saving text files in Jupyter (#561)
 - Test that notebooks paired through a configuration file are left unmodified (#598)
 - Test that metadata filters in the configuration files are taken into account when using `jupytext --to` (#543)
@@ -19,7 +21,7 @@ of R Markdown notebooks as a raw cell in Jupyter (#415)
 - The `# %%` cell marker has the same indentation as the first line in the cell (#562)
 - The `md:myst` and `md:pandoc` are always included in the Jupytext formats, and an informative runtime
 error will occur if the required dependencies, resp. `markdown-it-py` and `pandoc`, are not installed. (#556)
-- Jupytext now depends on `markdown-it-py` and always feature the MyST-Markdown format (Python 3.6 and above, #591)
+- Jupytext now depends on `markdown-it-py` and always features the MyST-Markdown format (Python 3.6 and above, #591)
 
 **Fixed**
 - Configured coverage targets in `codecov.yml`
diff --git a/jupytext/cell_to_text.py b/jupytext/cell_to_text.py
index 9a24122..2cc4d1d 100644
--- a/jupytext/cell_to_text.py
+++ b/jupytext/cell_to_text.py
@@ -44,7 +44,10 @@ class BaseCellExporter(object):
             cell.metadata, self.fmt.get("cell_metadata_filter"), _IGNORE_CELL_METADATA
         )
         if self.parse_cell_language:
-            self.language, magic_args = cell_language(self.source, default_language)
+            custom_cell_magics = self.fmt.get("custom_cell_magics", "").split(",")
+            self.language, magic_args = cell_language(
+                self.source, default_language, custom_cell_magics
+            )
 
             if magic_args:
                 self.metadata["magic_args"] = magic_args
diff --git a/jupytext/cli.py b/jupytext/cli.py
index 3b98867..828b663 100644
--- a/jupytext/cli.py
+++ b/jupytext/cli.py
@@ -404,7 +404,10 @@ def jupytext(args=None):
 def jupytext_single_file(nb_file, args, log):
     """Apply the jupytext commmand, with given arguments, to a single file"""
     if nb_file == "-" and args.sync:
-        raise ValueError("Cannot sync a notebook on stdin")
+        msg = "Missing notebook path."
+        if args.set_formats is not None and os.path.isfile(args.set_formats):
+            msg += " Maybe you mean 'jupytext --sync {}' ?".format(args.set_formats)
+        raise ValueError(msg)
 
     nb_dest = args.output or (
         None
diff --git a/jupytext/config.py b/jupytext/config.py
index 8c602de..177db55 100644
--- a/jupytext/config.py
+++ b/jupytext/config.py
@@ -136,6 +136,14 @@ class JupytextConfiguration(Configurable):
         config=True,
     )
 
+    custom_cell_magics = Unicode(
+        "",
+        help='A comma separated list of cell magics. Use e.g. custom_cell_magics = "configure,local" '
+        'if you want code cells starting with the Spark magic cell commands "configure" and "local" '
+        "to be commented out when converted to scripts.",
+        config=True,
+    )
+
     def set_default_format_options(self, format_options, read=False):
         """Set default format option"""
         if self.default_notebook_metadata_filter:
@@ -166,6 +174,8 @@ class JupytextConfiguration(Configurable):
             format_options.setdefault("cell_markers", self.default_cell_markers)
         if read and self.sphinx_convert_rst2md:
             format_options.setdefault("rst2md", self.sphinx_convert_rst2md)
+        if self.custom_cell_magics:
+            format_options.setdefault("custom_cell_magics", self.custom_cell_magics)
 
     def default_formats(self, path):
         """Return the default formats, if they apply to the current path #157"""
diff --git a/jupytext/formats.py b/jupytext/formats.py
index c7088bb..b1e6afd 100644
--- a/jupytext/formats.py
+++ b/jupytext/formats.py
@@ -709,6 +709,7 @@ _VALID_FORMAT_OPTIONS = _BINARY_FORMAT_OPTIONS + [
     "notebook_metadata_filter",
     "cell_metadata_filter",
     "cell_markers",
+    "custom_cell_magics",
 ]
 _VALID_FORMAT_NAMES = {fmt.format_name for fmt in JUPYTEXT_FORMATS}
 
diff --git a/jupytext/jupytext.py b/jupytext/jupytext.py
index 09fa4ee..740d62f 100644
--- a/jupytext/jupytext.py
+++ b/jupytext/jupytext.py
@@ -116,7 +116,10 @@ class TextNotebookConverter(NotebookReader, NotebookWriter):
                 )  # pragma: no cover
             lines = lines[pos:]
 
-        set_main_and_cell_language(metadata, cells, self.implementation.extension)
+        custom_cell_magics = self.fmt.get("custom_cell_magics", "").split(",")
+        set_main_and_cell_language(
+            metadata, cells, self.implementation.extension, custom_cell_magics
+        )
         cell_metadata = set()
         for cell in cells:
             cell_metadata.update(cell.metadata.keys())
diff --git a/jupytext/languages.py b/jupytext/languages.py
index 4560b5f..917effb 100644
--- a/jupytext/languages.py
+++ b/jupytext/languages.py
@@ -123,7 +123,7 @@ def same_language(kernel_language, language):
     return usual_language_name(kernel_language) == usual_language_name(language)
 
 
-def set_main_and_cell_language(metadata, cells, ext):
+def set_main_and_cell_language(metadata, cells, ext, custom_cell_magics):
     """Set main language for the given collection of cells, and
     use magics for cells that use other languages"""
     main_language = default_language_from_metadata_and_ext(metadata, ext)
@@ -152,7 +152,7 @@ def set_main_and_cell_language(metadata, cells, ext):
             if usual_language_name(language) == main_language:
                 continue
 
-            if language in _JUPYTER_LANGUAGES:
+            if language in _JUPYTER_LANGUAGES or language in custom_cell_magics:
                 cell["metadata"].pop("language")
                 magic = "%%" if main_language != "csharp" else "#!"
                 if "magic_args" in cell["metadata"]:
@@ -165,7 +165,7 @@ def set_main_and_cell_language(metadata, cells, ext):
                     cell["source"] = u"{}{}\n".format(magic, language) + cell["source"]
 
 
-def cell_language(source, default_language):
+def cell_language(source, default_language, custom_cell_magics):
     """Return cell language and language options, if any"""
     if source:
         line = source[0]
@@ -183,7 +183,7 @@ def cell_language(source, default_language):
                 lang = magic
                 magic_args = ""
 
-            if lang in _JUPYTER_LANGUAGES:
+            if lang in _JUPYTER_LANGUAGES or lang in custom_cell_magics:
                 source.pop(0)
                 return lang, magic_args
 


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 3169c0e..167fb5e 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -289,7 +289,6 @@ def test_combine_lower_version_raises(tmpdir):
         "This error occurs because notebook.py is in the light format in version 55.4, "
         "while jupytext version .* installed at .* can only read the light format in versions .*",
     ):
-
         jupytext([tmp_nbpy, "--to", "ipynb", "--update"])
 
 
@@ -1140,3 +1139,30 @@ def test_jupytext_to_file_emits_a_warning(tmpdir):
 
     with pytest.warns(UserWarning, match="Maybe you want to use the '-o' option"):
         jupytext(["notebook.ipynb", "--to", "script.py"])
+
+
+def test_jupytext_set_formats_file_gives_an_informative_error(tmpdir):
+    """The user may type
+        jupytext --set-formats notebook.md
+    meaning
+        jupytext --syn notebook.md
+    """
+    os.chdir(str(tmpdir))
+
+    cfg_file = tmpdir.join("jupytext.toml")
+    cfg_file.write('default_jupytext_formats = "md,ipynb,py:percent"')
+
+    md_file = tmpdir.join("notebook.md")
+    py_file = tmpdir.join("notebook.py")
+    nb_file = tmpdir.join("notebook.ipynb")
+    md_file.write("Some text")
+
+    with pytest.warns(None) as record:
+        jupytext(["--sync", "notebook.md"])
+
+    assert len(record) == 0
+    assert py_file.exists()
+    assert nb_file.exists()
+
+    with pytest.raises(ValueError, match="jupytext --sync notebook.md"):
+        jupytext(["--set-formats", "notebook.md"])
diff --git a/tests/test_custom_cell_magics.py b/tests/test_custom_cell_magics.py
new file mode 100644
index 0000000..97f39d4
--- /dev/null
+++ b/tests/test_custom_cell_magics.py
@@ -0,0 +1,41 @@
+import nbformat
+from nbformat.v4.nbbase import new_notebook, new_code_cell
+from jupytext.cli import jupytext as jupytext_cli
+from jupytext.languages import _JUPYTER_LANGUAGES_LOWER_AND_UPPER
+from jupytext.compare import compare_notebooks
+
+
+def test_custom_cell_magics(
+    tmpdir,
+    nb=new_notebook(
+        cells=[
+            new_code_cell("%%sql -o tables -q\n SHOW TABLES"),
+            new_code_cell(
+                """%%configure -f
+{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}"""
+            ),
+            new_code_cell("%%local\na=1"),
+        ]
+    ),
+):
+    cfg_file = tmpdir.join("jupytext.toml")
+    nb_file = tmpdir.join("notebook.ipynb")
+    py_file = tmpdir.join("notebook.py")
+
+    cfg_file.write('custom_cell_magics = "configure,local"')
+    assert "configure" not in _JUPYTER_LANGUAGES_LOWER_AND_UPPER
+    assert "logs" not in _JUPYTER_LANGUAGES_LOWER_AND_UPPER
+
+    nbformat.write(nb, str(nb_file))
+
+    jupytext_cli([str(nb_file), "--to", "py"])
+    py = py_file.read()
+
+    for line in py.splitlines():
+        if line:
+            assert line.startswith("# "), line
+
+    jupytext_cli([str(py_file), "--to", "notebook"])
+    nb2 = nbformat.read(str(nb_file), as_version=4)
+
+    compare_notebooks(nb2, nb)