Repository: mwouts/jupytext
Pull request #355 (2019-10-18 23:18:19)

Noeval attribute md
- Improve the compare_notebook function (show the difference in cell content, and catch a missing line break)
- Code cells with a Jupyter language, and a `.noeval` attribute, are not turned into code cells in Jupyter #347 

codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/355?src=pr&el=h1) Report
> Merging [#355](https://codecov.io/gh/mwouts/jupytext/pull/355?src=pr&el=desc) into [master](https://codecov.io/gh/mwouts/jupytext/commit/de1d536d648c8529192f62459c112e2660ee2496?src=pr&el=desc) will **decrease** coverage by `10.47%`.
> The diff coverage is `97.29%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/355/graphs/tree.svg?width=650&token=a1pNjsiILi&height=150&src=pr)](https://codecov.io/gh/mwouts/jupytext/pull/355?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master     #355       +/-   ##
===========================================
- Coverage   99.05%   88.58%   -10.48%     
===========================================
  Files          73       73               
  Lines        7086     6936      -150     
===========================================
- Hits         7019     6144      -875     
- Misses         67      792      +725
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/355?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jupytext/cell\_metadata.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-anVweXRleHQvY2VsbF9tZXRhZGF0YS5weQ==) | `90.15% <100%> (-7.2%)` | :arrow_down: |
| [jupytext/compare.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-anVweXRleHQvY29tcGFyZS5weQ==) | `99.34% <100%> (+0.01%)` | :arrow_up: |
| [tests/test\_read\_simple\_markdown.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9yZWFkX3NpbXBsZV9tYXJrZG93bi5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [jupytext/cell\_reader.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-anVweXRleHQvY2VsbF9yZWFkZXIucHk=) | `94.74% <93.33%> (-1.36%)` | :arrow_down: |
| [tests/test\_black.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9ibGFjay5weQ==) | `33.68% <0%> (-66.32%)` | :arrow_down: |
| [tests/test\_cell\_metadata.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jZWxsX21ldGFkYXRhLnB5) | `42.25% <0%> (-57.75%)` | :arrow_down: |
| [tests/test\_mirror.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9taXJyb3IucHk=) | `42.3% <0%> (-57.7%)` | :arrow_down: |
| [tests/test\_read\_simple\_pandoc.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9yZWFkX3NpbXBsZV9wYW5kb2MucHk=) | `44.82% <0%> (-55.18%)` | :arrow_down: |
| [jupytext/pandoc.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-anVweXRleHQvcGFuZG9jLnB5) | `40.81% <0%> (-55.11%)` | :arrow_down: |
| [tests/test\_knitr\_spin.py](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9rbml0cl9zcGluLnB5) | `45.45% <0%> (-54.55%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/mwouts/jupytext/pull/355/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/355?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/355?src=pr&el=footer). Last update [de1d536...85105fd](https://codecov.io/gh/mwouts/jupytext/pull/355?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/HISTORY.rst b/HISTORY.rst
index a5e63ba9..3bec69f5 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -11,6 +11,7 @@ Release History
 - Raw cells are now encoded using HTML comments (``<!-- #raw -->`` and ``<!-- #endraw -->``) in Markdown files (#321)
 - Markdown cells can be delimited with any of ``<!-- #region -->``,  ``<!-- #markdown -->`` or ``<!-- #md -->`` (#344)
 - Code blocks from Markdown files, when they don't have an explicit language, appear in Markdown cells in Jupyter (#321)
+- Code blocks with an explicit language and a ``.noeval`` attribute are inactive in Jupyter (#347)
 - Markdown and raw cells can be quoted with triple quotes in the ``py:percent`` format. And Markdown cells can start with just ``# %% [md]`` (#305)
 - ``jupytext notebook.py --to ipynb`` updates the timestamp of ``notebook.py`` so that the paired notebook still works in Jupyter (#335, #254)
 - The Jupyter Notebook extension for Jupytext is compatible with Jupyter Notebook 6.0 (#346)
diff --git a/jupytext/cell_metadata.py b/jupytext/cell_metadata.py
index 693f9ecf..4119246e 100644
--- a/jupytext/cell_metadata.py
+++ b/jupytext/cell_metadata.py
@@ -285,7 +285,7 @@ def is_active(ext, metadata, default=True):
             return ext.replace('.', '') in tag.split('-')
     if 'active' not in metadata:
         return default
-    return ext.replace('.', '') in re.split('\\.|,', metadata['active'])
+    return ext.replace('.', '') in re.split(r'\.|,', metadata['active'])
 
 
 def metadata_to_double_percent_options(metadata):
diff --git a/jupytext/cell_reader.py b/jupytext/cell_reader.py
index eee96d51..94839349 100644
--- a/jupytext/cell_reader.py
+++ b/jupytext/cell_reader.py
@@ -237,7 +237,7 @@ class MarkdownCellReader(BaseCellReader):
     """Read notebook cells from Markdown documents"""
     comment = ''
     start_code_re = re.compile(r"^```({})(.*)".format('|'.join(
-        _JUPYTER_LANGUAGES + [str.upper(lang) for lang in _JUPYTER_LANGUAGES]).replace('+', '\\+')))
+        _JUPYTER_LANGUAGES.union({str.upper(lang) for lang in _JUPYTER_LANGUAGES})).replace('+', '\\+')))
     non_jupyter_code_re = re.compile(r"^```")
     end_code_re = re.compile(r"^```\s*$")
     start_region_re = re.compile(r"^<!--\s*#(region|markdown|md|raw)(.*)-->\s*$")
@@ -270,6 +270,11 @@ class MarkdownCellReader(BaseCellReader):
                 self.metadata['region_name'] = region_name
         elif self.start_code_re.match(line):
             self.language, self.metadata = self.options_to_metadata(self.start_code_re.findall(line)[0])
+            # Cells with a .noeval attribute are markdown cells #347
+            if self.metadata.get('.noeval', '') is None:
+                self.cell_type = 'markdown'
+                self.metadata = {}
+                self.language = None
 
     def options_to_metadata(self, options):
         if isinstance(options, tuple):
@@ -306,12 +311,24 @@ class MarkdownCellReader(BaseCellReader):
                 if in_indented_code_block or in_explicit_code_block:
                     continue
 
-                if self.start_code_re.match(line) or self.start_region_re.match(line):
+                if self.start_region_re.match(line):
+                    if i > 1 and prev_blank:
+                        return i - 1, i, False
+                    return i, i, False
+
+                if self.start_code_re.match(line):
+                    # Cells with a .noeval attribute are markdown cells #347
+                    _, metadata = self.options_to_metadata(self.start_code_re.findall(line)[0])
+                    if metadata.get('.noeval', '') is None:
+                        in_explicit_code_block = True
+                        prev_blank = 0
+                        continue
+
                     if i > 1 and prev_blank:
                         return i - 1, i, False
                     return i, i, False
 
-                if self.non_jupyter_code_re and self.non_jupyter_code_re.match(line):
+                if self.non_jupyter_code_re.match(line):
                     in_explicit_code_block = True
                     prev_blank = 0
                     continue
diff --git a/jupytext/compare.py b/jupytext/compare.py
index 280d62e4..c765c4cd 100644
--- a/jupytext/compare.py
+++ b/jupytext/compare.py
@@ -16,20 +16,25 @@ _BLANK_LINE = re.compile(r'^\s*$')
 
 def _multilines(obj):
     try:
-        return obj.splitlines()
+        lines = obj.splitlines()
+        return lines + [''] if obj.endswith('\n') else lines
     except AttributeError:
         # Remove the final blank space on Python 2.7
         # return json.dumps(obj, indent=True, sort_keys=True).splitlines()
         return [line.rstrip() for line in json.dumps(obj, indent=True, sort_keys=True).splitlines()]
 
 
-def compare(actual, expected):
+def compare(actual, expected, return_diff=False):
     """Compare two strings, lists or dict-like objects"""
     if actual != expected:
-        raise AssertionError('\n' + '\n'.join(difflib.unified_diff(
+        diff = '\n'.join(difflib.unified_diff(
             _multilines(actual),
             _multilines(expected),
-            'first', 'second', lineterm='')))
+            'first', 'second', lineterm=''))
+        if return_diff:
+            return diff
+        raise AssertionError('\n' + diff)
+    return ''
 
 
 def filtered_cell(cell, preserve_outputs, cell_metadata_filter):
@@ -166,12 +171,11 @@ def compare_notebooks(notebook_actual, notebook_expected, fmt=None, allow_expect
 
         # 3. bis test entire cell content
         if not same_content(ref_cell.source, test_cell.source, allow_removed_final_blank_line):
-            try:
-                compare(ref_cell.source, test_cell.source)
-            except AssertionError as error:
+            if ref_cell.source != test_cell.source:
                 if raise_on_first_difference:
+                    diff = compare(ref_cell.source, test_cell.source, return_diff=True)
                     raise NotebookDifference("Cell content differ on {} cell #{}: {}"
-                                             .format(test_cell.cell_type, i, str(error)))
+                                             .format(test_cell.cell_type, i, diff))
                 modified_cells.add(i)
 
         if not compare_outputs:
diff --git a/jupytext/languages.py b/jupytext/languages.py
index 371bdfff..208a75a9 100644
--- a/jupytext/languages.py
+++ b/jupytext/languages.py
@@ -29,10 +29,7 @@ _COMMENT_CHARS = [_SCRIPT_EXTENSIONS[ext]['comment'] for ext in _SCRIPT_EXTENSIO
                   _SCRIPT_EXTENSIONS[ext]['comment'] != '#']
 
 _COMMENT = {_SCRIPT_EXTENSIONS[ext]['language']: _SCRIPT_EXTENSIONS[ext]['comment'] for ext in _SCRIPT_EXTENSIONS}
-
-_JUPYTER_LANGUAGES = _JUPYTER_LANGUAGES + [
-    _SCRIPT_EXTENSIONS[ext]['language'] for ext in _SCRIPT_EXTENSIONS if
-    _SCRIPT_EXTENSIONS[ext]['language'] not in _JUPYTER_LANGUAGES]
+_JUPYTER_LANGUAGES = set(_JUPYTER_LANGUAGES).union(_COMMENT.keys())
 
 
 def default_language_from_metadata_and_ext(metadata, ext):


Test patch:
diff --git a/tests/test_read_simple_markdown.py b/tests/test_read_simple_markdown.py
index 8fd514ff..1a49563d 100644
--- a/tests/test_read_simple_markdown.py
+++ b/tests/test_read_simple_markdown.py
@@ -455,3 +455,34 @@ def test_inactive_cell_using_tag(text='''```python tags=["active-md"]
     compare_notebooks(nb, expected)
     text2 = jupytext.writes(nb, 'md')
     compare(text2, text)
+
+
+def test_inactive_cell_using_noeval(text='''This is text
+
+```python .noeval
+# This is python code.
+# It should not become a code cell
+```
+'''):
+    expected = new_notebook(cells=[new_markdown_cell(text[:-1])])
+    nb = jupytext.reads(text, 'md')
+    compare_notebooks(nb, expected)
+    text2 = jupytext.writes(nb, 'md')
+    compare(text2, text)
+
+
+def test_noeval_followed_by_code_works(text='''```python .noeval
+# Not a code cell in Jupyter
+```
+
+```python
+1 + 1
+```
+''', expected=new_notebook(cells=[new_markdown_cell('''```python .noeval
+# Not a code cell in Jupyter
+```'''),
+                                  new_code_cell('1 + 1')])):
+    nb = jupytext.reads(text, 'md')
+    compare_notebooks(nb, expected)
+    text2 = jupytext.writes(nb, 'md')
+    compare(text2, text)