Repository: mwouts/jupytext

Issue:
configure magic conversion is seamless
%%configure magic code conversion is not seamless, for example:
```
%%configure -f
{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}

```
gets transformed to python code & r code
```
# %%configure -f
{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}

```
and gets transformed to scala code
```
// %%configure -f
{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}.

```

I think the conversion should be completely commented similar to SQL magic %%sql.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/608?src=pr&el=h1) Report
> Merging [#608](https://codecov.io/gh/mwouts/jupytext/pull/608?src=pr&el=desc) into [master](https://codecov.io/gh/mwouts/jupytext/commit/1f355fec4fa3a29b6d6a78f00dee27b05017d613?el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/608/graphs/tree.svg?width=650&height=150&src=pr&token=a1pNjsiILi)](https://codecov.io/gh/mwouts/jupytext/pull/608?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #608   +/-   ##
=======================================
  Coverage   98.98%   98.99%           
=======================================
  Files          92       93    +1     
  Lines        9090     9116   +26     
=======================================
+ Hits         8998     9024   +26     
  Misses         92       92           
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/608?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jupytext/formats.py](https://codecov.io/gh/mwouts/jupytext/pull/608/diff?src=pr&el=tree#diff-anVweXRleHQvZm9ybWF0cy5weQ==) | `96.91% <ø> (ø)` | |
| [jupytext/cell\_to\_text.py](https://codecov.io/gh/mwouts/jupytext/pull/608/diff?src=pr&el=tree#diff-anVweXRleHQvY2VsbF90b190ZXh0LnB5) | `97.41% <100.00%> (+<0.01%)` | :arrow_up: |
| [jupytext/config.py](https://codecov.io/gh/mwouts/jupytext/pull/608/diff?src=pr&el=tree#diff-anVweXRleHQvY29uZmlnLnB5) | `96.47% <100.00%> (+0.07%)` | :arrow_up: |
| [jupytext/jupytext.py](https://codecov.io/gh/mwouts/jupytext/pull/608/diff?src=pr&el=tree#diff-anVweXRleHQvanVweXRleHQucHk=) | `100.00% <100.00%> (ø)` | |
| [jupytext/languages.py](https://codecov.io/gh/mwouts/jupytext/pull/608/diff?src=pr&el=tree#diff-anVweXRleHQvbGFuZ3VhZ2VzLnB5) | `100.00% <100.00%> (ø)` | |
| [tests/test\_custom\_cell\_magics.py](https://codecov.io/gh/mwouts/jupytext/pull/608/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jdXN0b21fY2VsbF9tYWdpY3MucHk=) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/608?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/608?src=pr&el=footer). Last update [1f355fe...05000af](https://codecov.io/gh/mwouts/jupytext/pull/608?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 95dfc5a..f65d2a0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,10 +7,12 @@
 - New option `root_level_metadata_as_raw_cell`. Set it to `False` if you don't want to see root level metadata
 of R Markdown notebooks as a raw cell in Jupyter (#415)
 - New option `doxygen_equation_markers` to translate Markdown equations into Doxygen equations (#517)
+- New option `custom_cell_magics` to comment out cells starting with user-specific cell magics (#513)
 - Tested `isort` on notebooks (#553)
 - `jupytext notebook.ipynb --to filename.py` will warn that `--to` is used in place of `--output`.
 - Warn if 'Include Metadata' is off when saving text files in Jupyter (#561)
 - Test that notebooks paired through a configuration file are left unmodified (#598)
+- Test that metadata filters in the configuration files are taken into account when using `jupytext --to` (#543)
 
 **Changed**
 - Install Jupytext from source on MyBinder to avoid cache issues (#567)
@@ -18,7 +20,7 @@ of R Markdown notebooks as a raw cell in Jupyter (#415)
 - The `# %%` cell marker has the same indentation as the first line in the cell (#562)
 - The `md:myst` and `md:pandoc` are always included in the Jupytext formats, and an informative runtime
 error will occur if the required dependencies, resp. `markdown-it-py` and `pandoc`, are not installed. (#556)
-- Jupytext now depends on `markdown-it-py` and always feature the MyST-Markdown format (Python 3.6 and above, #591)
+- Jupytext now depends on `markdown-it-py` and always features the MyST-Markdown format (Python 3.6 and above, #591)
 
 **Fixed**
 - Configured coverage targets in `codecov.yml`
diff --git a/jupytext/cell_to_text.py b/jupytext/cell_to_text.py
index 9a24122..2cc4d1d 100644
--- a/jupytext/cell_to_text.py
+++ b/jupytext/cell_to_text.py
@@ -44,7 +44,10 @@ class BaseCellExporter(object):
             cell.metadata, self.fmt.get("cell_metadata_filter"), _IGNORE_CELL_METADATA
         )
         if self.parse_cell_language:
-            self.language, magic_args = cell_language(self.source, default_language)
+            custom_cell_magics = self.fmt.get("custom_cell_magics", "").split(",")
+            self.language, magic_args = cell_language(
+                self.source, default_language, custom_cell_magics
+            )
 
             if magic_args:
                 self.metadata["magic_args"] = magic_args
diff --git a/jupytext/config.py b/jupytext/config.py
index 8c602de..177db55 100644
--- a/jupytext/config.py
+++ b/jupytext/config.py
@@ -136,6 +136,14 @@ class JupytextConfiguration(Configurable):
         config=True,
     )
 
+    custom_cell_magics = Unicode(
+        "",
+        help='A comma separated list of cell magics. Use e.g. custom_cell_magics = "configure,local" '
+        'if you want code cells starting with the Spark magic cell commands "configure" and "local" '
+        "to be commented out when converted to scripts.",
+        config=True,
+    )
+
     def set_default_format_options(self, format_options, read=False):
         """Set default format option"""
         if self.default_notebook_metadata_filter:
@@ -166,6 +174,8 @@ class JupytextConfiguration(Configurable):
             format_options.setdefault("cell_markers", self.default_cell_markers)
         if read and self.sphinx_convert_rst2md:
             format_options.setdefault("rst2md", self.sphinx_convert_rst2md)
+        if self.custom_cell_magics:
+            format_options.setdefault("custom_cell_magics", self.custom_cell_magics)
 
     def default_formats(self, path):
         """Return the default formats, if they apply to the current path #157"""
diff --git a/jupytext/formats.py b/jupytext/formats.py
index c7088bb..b1e6afd 100644
--- a/jupytext/formats.py
+++ b/jupytext/formats.py
@@ -709,6 +709,7 @@ _VALID_FORMAT_OPTIONS = _BINARY_FORMAT_OPTIONS + [
     "notebook_metadata_filter",
     "cell_metadata_filter",
     "cell_markers",
+    "custom_cell_magics",
 ]
 _VALID_FORMAT_NAMES = {fmt.format_name for fmt in JUPYTEXT_FORMATS}
 
diff --git a/jupytext/jupytext.py b/jupytext/jupytext.py
index 09fa4ee..740d62f 100644
--- a/jupytext/jupytext.py
+++ b/jupytext/jupytext.py
@@ -116,7 +116,10 @@ class TextNotebookConverter(NotebookReader, NotebookWriter):
                 )  # pragma: no cover
             lines = lines[pos:]
 
-        set_main_and_cell_language(metadata, cells, self.implementation.extension)
+        custom_cell_magics = self.fmt.get("custom_cell_magics", "").split(",")
+        set_main_and_cell_language(
+            metadata, cells, self.implementation.extension, custom_cell_magics
+        )
         cell_metadata = set()
         for cell in cells:
             cell_metadata.update(cell.metadata.keys())
diff --git a/jupytext/languages.py b/jupytext/languages.py
index 4560b5f..917effb 100644
--- a/jupytext/languages.py
+++ b/jupytext/languages.py
@@ -123,7 +123,7 @@ def same_language(kernel_language, language):
     return usual_language_name(kernel_language) == usual_language_name(language)
 
 
-def set_main_and_cell_language(metadata, cells, ext):
+def set_main_and_cell_language(metadata, cells, ext, custom_cell_magics):
     """Set main language for the given collection of cells, and
     use magics for cells that use other languages"""
     main_language = default_language_from_metadata_and_ext(metadata, ext)
@@ -152,7 +152,7 @@ def set_main_and_cell_language(metadata, cells, ext):
             if usual_language_name(language) == main_language:
                 continue
 
-            if language in _JUPYTER_LANGUAGES:
+            if language in _JUPYTER_LANGUAGES or language in custom_cell_magics:
                 cell["metadata"].pop("language")
                 magic = "%%" if main_language != "csharp" else "#!"
                 if "magic_args" in cell["metadata"]:
@@ -165,7 +165,7 @@ def set_main_and_cell_language(metadata, cells, ext):
                     cell["source"] = u"{}{}\n".format(magic, language) + cell["source"]
 
 
-def cell_language(source, default_language):
+def cell_language(source, default_language, custom_cell_magics):
     """Return cell language and language options, if any"""
     if source:
         line = source[0]
@@ -183,7 +183,7 @@ def cell_language(source, default_language):
                 lang = magic
                 magic_args = ""
 
-            if lang in _JUPYTER_LANGUAGES:
+            if lang in _JUPYTER_LANGUAGES or lang in custom_cell_magics:
                 source.pop(0)
                 return lang, magic_args
 


Test patch:
diff --git a/tests/test_custom_cell_magics.py b/tests/test_custom_cell_magics.py
new file mode 100644
index 0000000..97f39d4
--- /dev/null
+++ b/tests/test_custom_cell_magics.py
@@ -0,0 +1,41 @@
+import nbformat
+from nbformat.v4.nbbase import new_notebook, new_code_cell
+from jupytext.cli import jupytext as jupytext_cli
+from jupytext.languages import _JUPYTER_LANGUAGES_LOWER_AND_UPPER
+from jupytext.compare import compare_notebooks
+
+
+def test_custom_cell_magics(
+    tmpdir,
+    nb=new_notebook(
+        cells=[
+            new_code_cell("%%sql -o tables -q\n SHOW TABLES"),
+            new_code_cell(
+                """%%configure -f
+{"executorMemory": "3072M", "executorCores": 4, "numExecutors":10}"""
+            ),
+            new_code_cell("%%local\na=1"),
+        ]
+    ),
+):
+    cfg_file = tmpdir.join("jupytext.toml")
+    nb_file = tmpdir.join("notebook.ipynb")
+    py_file = tmpdir.join("notebook.py")
+
+    cfg_file.write('custom_cell_magics = "configure,local"')
+    assert "configure" not in _JUPYTER_LANGUAGES_LOWER_AND_UPPER
+    assert "logs" not in _JUPYTER_LANGUAGES_LOWER_AND_UPPER
+
+    nbformat.write(nb, str(nb_file))
+
+    jupytext_cli([str(nb_file), "--to", "py"])
+    py = py_file.read()
+
+    for line in py.splitlines():
+        if line:
+            assert line.startswith("# "), line
+
+    jupytext_cli([str(py_file), "--to", "notebook"])
+    nb2 = nbformat.read(str(nb_file), as_version=4)
+
+    compare_notebooks(nb2, nb)
diff --git a/tests/test_metadata_filters_from_config.py b/tests/test_metadata_filters_from_config.py
new file mode 100644
index 0000000..8efb7e3
--- /dev/null
+++ b/tests/test_metadata_filters_from_config.py
@@ -0,0 +1,48 @@
+import nbformat
+from nbformat.v4.nbbase import new_notebook, new_markdown_cell
+from jupytext.cli import jupytext as jupytext_cli
+from jupytext.compare import compare, compare_notebooks
+
+
+def test_metadata_filters_from_config(tmpdir):
+    cfg_file = tmpdir.join("jupytext.toml")
+    nb_file = tmpdir.join("notebook.ipynb")
+    md_file = tmpdir.join("notebook.md")
+
+    cfg_file.write(
+        """default_notebook_metadata_filter = "-all"
+default_cell_metadata_filter = "-all"
+"""
+    )
+    nb = new_notebook(
+        cells=[new_markdown_cell("A markdown cell")],
+        metadata={
+            "kernelspec": {
+                "display_name": "Python [conda env:.conda-week1]",
+                "language": "python",
+                "name": "conda-env-.conda-week1-py",
+            },
+            "language_info": {
+                "codemirror_mode": {"name": "ipython", "version": 3},
+                "file_extension": ".py",
+                "mimetype": "text/x-python",
+                "name": "python",
+                "nbconvert_exporter": "python",
+                "pygments_lexer": "ipython3",
+                "version": "3.8.3",
+            },
+            "nbsphinx": {"execute": "never"},
+        },
+    )
+    nbformat.write(nb, str(nb_file))
+
+    jupytext_cli([str(nb_file), "--to", "md"])
+    md = md_file.read()
+
+    compare(md, "A markdown cell\n")
+
+    jupytext_cli([str(md_file), "--to", "notebook", "--update"])
+    nb2 = nbformat.read(str(nb_file), as_version=4)
+
+    del nb2.metadata["jupytext"]
+    compare_notebooks(nb2, nb)