Repository: mwouts/jupytext

Issue:
Jupytext should not remove metadata from the YAML header in text files
The file `funnel-charts.md` contains a `plotly` entry in the YAML header, but a round trip in Jupytext [removes it](https://github.com/mwouts/plotly.py-docs/commit/335c5d1ceea263d36da2cfe895fcf3c6b901feeb#diff-79a654256c6b0d4336e702730725dc0c).

It would be cleaner if Jupytext could add a filter to make sure that the `plotly` section remains there.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/380?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@c189035`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/380/graphs/tree.svg?width=650&token=a1pNjsiILi&height=150&src=pr)](https://codecov.io/gh/mwouts/jupytext/pull/380?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master     #380   +/-   ##
=========================================
  Coverage          ?   99.06%           
=========================================
  Files             ?       75           
  Lines             ?     7453           
  Branches          ?        0           
=========================================
  Hits              ?     7383           
  Misses            ?       70           
  Partials          ?        0
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/380?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jupytext/header.py](https://codecov.io/gh/mwouts/jupytext/pull/380/diff?src=pr&el=tree#diff-anVweXRleHQvaGVhZGVyLnB5) | `99.23% <100%> (ø)` | |
| [jupytext/metadata\_filter.py](https://codecov.io/gh/mwouts/jupytext/pull/380/diff?src=pr&el=tree#diff-anVweXRleHQvbWV0YWRhdGFfZmlsdGVyLnB5) | `100% <100%> (ø)` | |
| [tests/test\_read\_simple\_markdown.py](https://codecov.io/gh/mwouts/jupytext/pull/380/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9yZWFkX3NpbXBsZV9tYXJrZG93bi5weQ==) | `100% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/380?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/380?src=pr&el=footer). Last update [c189035...44a2115](https://codecov.io/gh/mwouts/jupytext/pull/380?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8194baf..832e40b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,8 +13,9 @@
 - The light format uses `# + [markdown]` rather than the previous `cell_type` metadata to identify markdown cells with metadata (#356)
 - Explicit Markdown cells in the light format `# + [markdown]` can use triple quotes (#356)
 - IPython magic help commands like `float?` are classified as magics, and thus commented in Python scripts (#297)
-- Cell metadata can be encoded as either key=value (the new default) or in JSON. An automatic option `cell_metadata_json` should help minimize the impact on existing files (#344)
+- Cell metadata can be encoded as either `key=value` (the new default) or in JSON. An automatic option `cell_metadata_json` should help minimize the impact on existing files (#344)
 - R Markdown hidden inputs, outputs, or cells are now mapped to the corresponding Jupyter Book tags by default (#337)
+- The notebook metadata filter is automatically updated when extra keys are added to the YAML header (#376)
 - The Jupyter Notebook extension for Jupytext is compatible with Jupyter Notebook 6.0 (#346)
 - `jupytext notebook.py --to ipynb` updates the timestamp of `notebook.py` so that the paired notebook still works in Jupyter (#335, #254)
 - `jupytext --check pytest notebook.ipynb` can be used to run test functions in a notebook (#286)
diff --git a/jupytext/header.py b/jupytext/header.py
index c2933f3..b477b55 100644
--- a/jupytext/header.py
+++ b/jupytext/header.py
@@ -8,7 +8,7 @@ import nbformat
 from nbformat.v4.nbbase import new_raw_cell
 from .version import __version__
 from .languages import _SCRIPT_EXTENSIONS, comment_lines
-from .metadata_filter import filter_metadata
+from .metadata_filter import filter_metadata, _DEFAULT_NOTEBOOK_METADATA
 from .pep8 import pep8_lines_between_cells
 
 SafeRepresenter.add_representer(nbformat.NotebookNode, SafeRepresenter.represent_dict)
@@ -19,14 +19,6 @@ _JUPYTER_RE = re.compile(r"^jupyter\s*:\s*$")
 _LEFTSPACE_RE = re.compile(r"^\s")
 _UTF8_HEADER = ' -*- coding: utf-8 -*-'
 
-_DEFAULT_NOTEBOOK_METADATA = ','.join([
-    # Preserve Jupytext section
-    'jupytext',
-    # Preserve kernel specs
-    'kernelspec',
-    # Kernel_info found in Nteract notebooks
-    'kernel_info'])
-
 # Change this to False in tests
 INSERT_AND_CHECK_VERSION_NUMBER = True
 
diff --git a/jupytext/metadata_filter.py b/jupytext/metadata_filter.py
index 989f567..fecb28a 100644
--- a/jupytext/metadata_filter.py
+++ b/jupytext/metadata_filter.py
@@ -2,6 +2,14 @@
 
 from .cell_metadata import _JUPYTEXT_CELL_METADATA
 
+_DEFAULT_NOTEBOOK_METADATA = ','.join([
+    # Preserve Jupytext section
+    'jupytext',
+    # Preserve kernel specs
+    'kernelspec',
+    # Kernel_info found in Nteract notebooks
+    'kernel_info'])
+
 
 def metadata_filter_as_dict(metadata_config):
     """Return the metadata filter represented as either None (no filter),
@@ -84,6 +92,18 @@ def update_metadata_filters(metadata, jupyter_md, cell_metadata):
         cell_metadata = {'additional': cell_metadata, 'excluded': 'all'}
         metadata.setdefault('jupytext', {})['notebook_metadata_filter'] = '-all'
         metadata.setdefault('jupytext', {})['cell_metadata_filter'] = metadata_filter_as_string(cell_metadata)
+    else:
+        # Update the notebook metadata filter to include existing entries 376
+        nb_md_filter = metadata.get('jupytext', {}).get('notebook_metadata_filter', '').split(',')
+        nb_md_filter = [key for key in nb_md_filter if key]
+        if 'all' in nb_md_filter or '-all' in nb_md_filter:
+            return
+        for key in metadata:
+            if key in _DEFAULT_NOTEBOOK_METADATA or key in nb_md_filter or ('-' + key) in nb_md_filter:
+                continue
+            nb_md_filter.append(key)
+        if nb_md_filter:
+            metadata.setdefault('jupytext', {})['notebook_metadata_filter'] = ','.join(nb_md_filter)
 
 
 def apply_metadata_filters(user_filter, default_filter, actual_keys):


Test patch:
diff --git a/tests/test_read_simple_markdown.py b/tests/test_read_simple_markdown.py
index 8e9c38e..711206b 100644
--- a/tests/test_read_simple_markdown.py
+++ b/tests/test_read_simple_markdown.py
@@ -595,9 +595,9 @@ def test_two_markdown_cell_with_no_language_code_works(nb=new_notebook(cells=[
 def test_notebook_with_python3_magic(no_jupytext_version_number,
                                      nb=new_notebook(metadata={
                                          'kernelspec': {'display_name': 'Python 3', 'language': 'python',
-                                                        'name': 'python3'}},
-                                                     cells=[new_code_cell('%%python2\na = 1\nprint a'),
-                                                            new_code_cell('%%python3\nb = 2\nprint(b)')]),
+                                                        'name': 'python3'}}, cells=[
+                                         new_code_cell('%%python2\na = 1\nprint a'),
+                                         new_code_cell('%%python3\nb = 2\nprint(b)')]),
                                      text="""---
 jupyter:
   kernelspec:
@@ -621,3 +621,59 @@ print(b)
 
     nb2 = jupytext.reads(md, 'md')
     compare_notebooks(nb2, nb)
+
+
+def test_update_metadata_filter(
+        no_jupytext_version_number,
+        org="""---
+jupyter:
+  kernelspec:
+    display_name: Python 3
+    language: python
+    name: python3
+  extra:
+    key: value
+---
+""", target="""---
+jupyter:
+  extra:
+    key: value
+  jupytext:
+    notebook_metadata_filter: extra
+  kernelspec:
+    display_name: Python 3
+    language: python
+    name: python3
+---
+"""):
+    nb = jupytext.reads(org, 'md')
+    text = jupytext.writes(nb, 'md')
+    compare(text, target)
+
+
+def test_update_metadata_filter_2(
+        no_jupytext_version_number,
+        org="""---
+jupyter:
+  jupytext:
+    notebook_metadata_filter: -extra
+  kernelspec:
+    display_name: Python 3
+    language: python
+    name: python3
+  extra:
+    key: value
+---
+""", target="""---
+jupyter:
+  jupytext:
+    notebook_metadata_filter: -extra
+  kernelspec:
+    display_name: Python 3
+    language: python
+    name: python3
+---
+"""):
+    nb = jupytext.reads(org, 'md')
+    text = jupytext.writes(nb, 'md')
+    compare(text, target)