Repository: mwouts/jupytext
Pull request #371 (2019-11-02 16:01:03)

Jupytext pytest

codecov[bot]: # [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/371?src=pr&el=h1) Report
> Merging [#371](https://codecov.io/gh/mwouts/jupytext/pull/371?src=pr&el=desc) into [master](https://codecov.io/gh/mwouts/jupytext/commit/7ef84eeb8360f7e643e61ad52b7b6dc87686c153?src=pr&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `92.3%`.

[![Impacted file tree graph](https://codecov.io/gh/mwouts/jupytext/pull/371/graphs/tree.svg?width=650&token=a1pNjsiILi&height=150&src=pr)](https://codecov.io/gh/mwouts/jupytext/pull/371?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #371      +/-   ##
==========================================
- Coverage   99.05%   98.88%   -0.17%     
==========================================
  Files          74       75       +1     
  Lines        7316     7197     -119     
==========================================
- Hits         7247     7117     -130     
- Misses         69       80      +11
```


| [Impacted Files](https://codecov.io/gh/mwouts/jupytext/pull/371?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [tests/test\_cli.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9jbGkucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [jupytext/compare.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-anVweXRleHQvY29tcGFyZS5weQ==) | `99.34% <100%> (+0.64%)` | :arrow_up: |
| [jupytext/formats.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-anVweXRleHQvZm9ybWF0cy5weQ==) | `98.64% <100%> (-0.34%)` | :arrow_down: |
| [tests/test\_black.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9ibGFjay5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [tests/test\_pytest.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9weXRlc3QucHk=) | `100% <100%> (ø)` | |
| [jupytext/cli.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-anVweXRleHQvY2xpLnB5) | `95.48% <87.5%> (-0.86%)` | :arrow_down: |
| [jupytext/pep8.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-anVweXRleHQvcGVwOC5weQ==) | `95.23% <0%> (-4.77%)` | :arrow_down: |
| [jupytext/jupytext.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-anVweXRleHQvanVweXRleHQucHk=) | `98.62% <0%> (-1.38%)` | :arrow_down: |
| [jupytext/cell\_reader.py](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree#diff-anVweXRleHQvY2VsbF9yZWFkZXIucHk=) | `96.73% <0%> (-0.19%)` | :arrow_down: |
| ... and [28 more](https://codecov.io/gh/mwouts/jupytext/pull/371/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/mwouts/jupytext/pull/371?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/mwouts/jupytext/pull/371?src=pr&el=footer). Last update [7ef84ee...8da8879](https://codecov.io/gh/mwouts/jupytext/pull/371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/HISTORY.rst b/HISTORY.rst
index 2cc996be..aabe4f5d 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -19,6 +19,8 @@ Release History
 - R Markdown hidden inputs, outputs, or cells are now mapped to the corresponding Jupyter Book tags by default (#337)
 - The Jupyter Notebook extension for Jupytext is compatible with Jupyter Notebook 6.0 (#346)
 - ``jupytext notebook.py --to ipynb`` updates the timestamp of ``notebook.py`` so that the paired notebook still works in Jupyter (#335, #254)
+- ``jupytext --check pytest notebook.ipynb`` can be used to run test functions in a notebook (#286)
+- ``jupytext --check`` and ``jupytext --pipe`` can run commands that only operate on files: when ``{}`` is found in the text of the command, ``jupytext`` saves the text representation of the notebook in a temp file, and replaces ``{}`` with the name of that file before executing the command. (#286)
 - Added support for Rust/Evxcr, by Jonas Bushart (#351)
 - Added support for PowerShell (#349)
 
diff --git a/docs/using-cli.md b/docs/using-cli.md
index 8aa0c17f..df336ad7 100644
--- a/docs/using-cli.md
+++ b/docs/using-cli.md
@@ -42,8 +42,13 @@ You may also find useful to `--pipe` the text representation of a notebook into
 jupytext --sync --pipe black notebook.ipynb    # read most recent version of notebook, reformat with black, save
 ```
 
-Execute `jupytext --help` to access the full documentation.
+For programs that don't accept pipes, use `{}` as a placeholder for the name of a temporary file that will contain the text representation of the notebook. For instance, run `pytest` on your notebook with:
+```bash
+jupytext --check 'pytest {}' notebook.ipynb    # export the notebook in format py:percent in a temp file, run pytest
+```
+(read more about running `pytest` on notebooks in our example [`Tests in a notebook.md`](https://github.com/mwouts/jupytext/blob/master/demo/Tests%20in%20a%20notebook.md)).
 
+Execute `jupytext --help` to access the full documentation.
 
 ## Notebook and cell metadata filters
 
diff --git a/jupytext/cli.py b/jupytext/cli.py
index 5d42ffaa..99973708 100644
--- a/jupytext/cli.py
+++ b/jupytext/cli.py
@@ -8,6 +8,7 @@ import subprocess
 import argparse
 import json
 from copy import copy
+from tempfile import NamedTemporaryFile
 from .jupytext import read, reads, write, writes
 from .formats import _VALID_FORMAT_OPTIONS, _BINARY_FORMAT_OPTIONS, check_file_version
 from .formats import long_form_one_format, long_form_multiple_formats, short_form_one_format, check_auto_ext
@@ -157,18 +158,28 @@ def parse_jupytext_args(args=None):
     # Pipe notebook inputs into other commands
     parser.add_argument('--pipe',
                         action='append',
-                        help='Pipe the text representation of the notebook into another\n'
-                             'program, and read the notebook back. For instance, reformat\n'
+                        help='Pipe the text representation (in format --pipe-fmt) of the notebook into\n'
+                             'another program, and read the notebook back. For instance, reformat\n'
                              'your notebook with:\n'
-                             "    jupytext notebook.ipynb --pipe black\n"
+                             '    jupytext notebook.ipynb --pipe black\n'
                              'If you want to reformat it and sync the paired representation, execute:\n'
-                             "    jupytext notebook.ipynb --sync --pipe black\n")
+                             '    jupytext notebook.ipynb --sync --pipe black\n'
+                             'In case the program that you want to execute does not accept pipes, use {}\n'
+                             'as a placeholder for a temporary file name into which jupytext will\n'
+                             'write the text representation of the notebook, e.g.:\n'
+                             "    jupytext notebook.ipynb --pipe 'black {}'\n")
     parser.add_argument('--check',
                         action='append',
-                        help='Pipe the text representation of the notebook into another program,\n'
-                             'and test that the returned value is non zero. For instance,\n'
-                             'test that your notebook is pep8 compliant with:\n'
-                             "    jupytext notebook.ipynb --check flake8\n")
+                        help='Pipe the text representation (in format --pipe-fmt) of the notebook into\n'
+                             'another program, and test that the returned value is non zero. For\n'
+                             'instance, test that your notebook is pep8 compliant with:\n'
+                             '    jupytext notebook.ipynb --check flake8\n'
+                             'or run pytest on your notebook with:\n'
+                             '    jupytext notebook.ipynb --check pytest\n'
+                             'In case the program that you want to execute does not accept pipes, use {}\n'
+                             'as a placeholder for a temporary file name into which jupytext will\n'
+                             'write the text representation of the notebook, e.g.:\n'
+                             "    jupytext notebook.ipynb --check 'pytest {}'\n")
     parser.add_argument('--pipe-fmt',
                         default='auto:percent',
                         help='The format in which the notebook should be piped to other programs,\n'
@@ -360,12 +371,13 @@ def jupytext_single_file(nb_file, args, log):
 
     # II. ### Apply commands onto the notebook ###
     # Pipe the notebook into the desired commands
+    prefix = None if nb_file == '-' else os.path.splitext(os.path.basename(nb_file))[0]
     for cmd in args.pipe or []:
-        notebook = pipe_notebook(notebook, cmd, args.pipe_fmt)
+        notebook = pipe_notebook(notebook, cmd, args.pipe_fmt, prefix=prefix)
 
     # and/or test the desired commands onto the notebook
     for cmd in args.check or []:
-        pipe_notebook(notebook, cmd, args.pipe_fmt, update=False)
+        pipe_notebook(notebook, cmd, args.pipe_fmt, update=False, prefix=prefix)
 
     # Execute the notebook
     if args.execute:
@@ -568,31 +580,73 @@ def load_paired_notebook(notebook, fmt, nb_file, log):
     return notebook, latest_inputs, latest_outputs
 
 
-def pipe_notebook(notebook, command, fmt='py:percent', update=True):
+def exec_command(command, input=None):
+    """Execute the desired command, and pipe the given input into it"""
+    if not isinstance(command, list):
+        command = command.split(' ')
+    process = subprocess.Popen(command,
+                               **(dict(stdout=subprocess.PIPE, stdin=subprocess.PIPE) if input is not None else {}))
+    out, err = process.communicate(input=input)
+
+    if process.returncode:
+        sys.stderr.write("The command {} exited with code {}{}"
+                         .format(command, process.returncode, ': {}'.format(err or out) if err or out else ''))
+        raise SystemExit(process.returncode)
+
+    return out
+
+
+def pipe_notebook(notebook, command, fmt='py:percent', update=True, prefix=None):
     """Pipe the notebook, in the desired representation, to the given command. Update the notebook
     with the returned content if desired."""
     if command in ['black', 'flake8', 'autopep8']:
         command = command + ' -'
+    elif command in ['pytest', 'unittest']:
+        command = command + ' {}'
 
     fmt = long_form_one_format(fmt, notebook.metadata, auto_ext_requires_language_info=False)
     check_auto_ext(fmt, notebook.metadata, '--pipe-fmt')
     text = writes(notebook, fmt)
-    process = subprocess.Popen(command.split(' '), stdout=subprocess.PIPE, stdin=subprocess.PIPE)
-    cmd_output, err = process.communicate(input=text.encode('utf-8'))
 
-    if process.returncode:
-        sys.stderr.write("Command '{}' exited with code {}: {}"
-                         .format(command, process.returncode, err or cmd_output))
-        raise SystemExit(process.returncode)
+    command = command.split(' ')
+    if '{}' in command:
+        if prefix is not None:
+            prefix = prefix + (' ' if ' ' in prefix else '_')
+        tmp_file_args = dict(mode='w+',
+                             encoding='utf8',
+                             prefix=prefix,
+                             suffix=fmt['extension'],
+                             delete=False)
+        try:
+            tmp = NamedTemporaryFile(**tmp_file_args)
+        except TypeError:
+            # NamedTemporaryFile does not have an 'encoding' argument on pypy
+            tmp_file_args.pop('encoding')
+            tmp = NamedTemporaryFile(**tmp_file_args)
+        try:
+            tmp.write(text)
+            tmp.close()
+
+            exec_command([cmd if cmd != '{}' else tmp.name for cmd in command])
+
+            if not update:
+                return notebook
+
+            piped_notebook = read(tmp.name, fmt=fmt)
+        finally:
+            os.remove(tmp.name)
+    else:
+        cmd_output = exec_command(command, text.encode('utf-8'))
 
-    if not update:
-        return notebook
+        if not update:
+            return notebook
 
-    if not cmd_output:
-        sys.stderr.write("[jupytext] The command '{}' had no output. As a result, the notebook is empty. "
-                         "Is this expected? If not, use --check rather than --pipe for this command.".format(command))
+        if not cmd_output:
+            sys.stderr.write("[jupytext] The command '{}' had no output. As a result, the notebook is empty. "
+                             "Is this expected? If not, use --check rather than --pipe for this command."
+                             .format(command))
 
-    piped_notebook = reads(cmd_output.decode('utf-8'), fmt)
+        piped_notebook = reads(cmd_output.decode('utf-8'), fmt)
 
     if fmt['extension'] != '.ipynb':
         combine_inputs_with_outputs(piped_notebook, notebook, fmt)
diff --git a/jupytext/compare.py b/jupytext/compare.py
index 885b3dcf..c488ef0e 100644
--- a/jupytext/compare.py
+++ b/jupytext/compare.py
@@ -30,12 +30,11 @@ def compare(actual, expected, return_diff=False):
         diff = '\n'.join(difflib.unified_diff(
             _multilines(expected),
             _multilines(actual),
-            'actual', 'expected', lineterm=''))
+            'expected', 'actual', lineterm=''))
         if return_diff:
             return diff
         raise AssertionError('\n' + diff)
-    if return_diff:
-        return ''
+    return '' if return_diff else None
 
 
 def filtered_cell(cell, preserve_outputs, cell_metadata_filter):
diff --git a/jupytext/formats.py b/jupytext/formats.py
index f28494ce..7e0c09b2 100644
--- a/jupytext/formats.py
+++ b/jupytext/formats.py
@@ -567,10 +567,10 @@ def auto_ext_from_metadata(metadata):
     auto_ext = metadata.get('language_info', {}).get('file_extension')
 
     if auto_ext is None:
-        kernel_language = metadata.get('kernelspec', {}).get('language')
-        if kernel_language:
+        language = metadata.get('kernelspec', {}).get('language') or metadata.get('jupytext', {}).get('main_language')
+        if language:
             for ext in _SCRIPT_EXTENSIONS:
-                if same_language(kernel_language, _SCRIPT_EXTENSIONS[ext]['language']):
+                if same_language(language, _SCRIPT_EXTENSIONS[ext]['language']):
                     auto_ext = ext
                     break
 


Test patch:
diff --git a/demo/Tests in a notebook.md b/demo/Tests in a notebook.md
new file mode 100644
index 00000000..ab4cfef1
--- /dev/null
+++ b/demo/Tests in a notebook.md
@@ -0,0 +1,63 @@
+# Testing a Jupyter notebook with pytest
+
+In this notebook we describe how to test a notebook with `jupytext`.
+
+## Writing assertions and tests in a notebook
+
+Our notebook defines a function that we wish to test. Our function is simply
+
+```python
+def f(x, n=5):
+    return [x + i for i in range(n)] 
+```
+
+We can test the assertion in Jupyter with simply
+
+```python
+assert f(5) == [5,6,7,8,9]
+```
+
+Since the assertion above works, we don't get any message. It's more interesting to see what happens when an assertion fails. Remove one element of the list above and change the assertion to, say, 
+
+    assert f(5) == [5,6,8,9]
+
+When we run the above in Jupyter, we get
+
+```stderr
+---------------------------------------------------------------------------
+AssertionError                            Traceback (most recent call last)
+<ipython-input-3-1383ac5d204f> in <module>
+----> 1 assert f(5) == [5,6,8,9]
+
+AssertionError: 
+```
+
+Now if we run the notebook with `jupytext --check pytest 'Tests in a notebook.md'`, we get a more detailed description of the issue, thanks to `pytest`'s rewriting of assertions: 
+```output
+[jupytext] Reading Tests in a notebook.md
+=========================== test session starts ===========================
+platform win32 -- Python 3.7.5, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
+rootdir: C:\Users\Marco
+collected 0 items / 1 errors
+
+================================ ERRORS ===================================
+_________ ERROR collecting Tests in a notebook vhs_lscr.py ________________
+Tests in a notebook vhs_lscr.py:19: in <module>
+    assert f(5) == [5,6,8,9]
+E   assert [5, 6, 7, 8, 9] == [5, 6, 8, 9]
+E    +  where [5, 6, 7, 8, 9] = <function f at 0x000002440A0D1798>(5)
+!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!
+=========================== 1 error in 0.09s ==============================
+```
+
+Once all of our assertions pass, we can move them to a test function. In Jupyter the function is not evaluated - only when we run `jupytext --check pytest` on the notebook, the function is actually executed.
+
+```python
+def test_f():
+    assert f(5) == [5,6,7,8,9]
+```
+
+## Going further
+
+- [nbval](https://github.com/computationalmodelling/nbval) is a plugin for `pytest` that allows you to make sure that Jupyter notebooks run properly, and that their new outputs match the current ones. Use it as `pytest --nbval notebook.ipynb`.
+- [ipytest](https://github.com/chmp/ipytest) defines a `%%run_pytest` cell magic that allows you to execute the tests in a cell directly in Jupyter.
diff --git a/tests/test_black.py b/tests/test_black.py
index 5bef4c5c..5a2190e3 100644
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -187,3 +187,23 @@ jupyter:
 
     nb = read(tmp_md)
     compare(nb.cells, [new_code_cell('1 + 2 + 3 + 4')])
+
+
+@requires_black
+def test_black_through_tempfile(
+        tmpdir,
+        text="""```python
+1 +    2 \
++ 3
+```
+""",
+        black="""```python
+1 + 2 + 3
+```
+"""):
+    tmp_md = str(tmpdir.join('notebook.md'))
+    with open(tmp_md, 'w') as fp:
+        fp.write(text)
+    jupytext([tmp_md, '--pipe', 'black {}'])
+    with open(tmp_md) as fp:
+        compare(fp.read(), black)
diff --git a/tests/test_cli.py b/tests/test_cli.py
index c2052b47..c2440aa7 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -990,17 +990,3 @@ def test_339_require_to(tmpdir):
 
     with pytest.raises(ValueError, match='--to'):
         jupytext([tmp_py, '--test-strict'])
-
-
-@requires_black
-def test_detailed_message_missing_language_info(tmpdir):
-    text = """```python
-1 + 1
-```
-"""
-    tmp_md = str(tmpdir.join('test.md'))
-    with open(tmp_md, 'w') as fp:
-        fp.write(text)
-
-    with pytest.raises(ValueError, match='--pipe-fmt'):
-        jupytext([tmp_md, '--pipe', 'black'])
diff --git a/tests/test_compare.py b/tests/test_compare.py
index cf300876..2e8e5686 100644
--- a/tests/test_compare.py
+++ b/tests/test_compare.py
@@ -64,8 +64,8 @@ def test_compare_on_notebooks(notebook_actual, notebook_expected):
         compare(notebook_actual, notebook_expected)
 
     assert str(err.value) == """
---- actual
-+++ expected
+--- expected
++++ actual
 @@ -15,7 +15,7 @@
    {
     "cell_type": "markdown",
diff --git a/tests/test_pytest.py b/tests/test_pytest.py
new file mode 100644
index 00000000..eaf3ea95
--- /dev/null
+++ b/tests/test_pytest.py
@@ -0,0 +1,32 @@
+import pytest
+from jupytext.cli import jupytext
+
+
+def test_run_pytest_ok(
+        tmpdir,
+        text="""This is a notebook with a test that is supposed to pass
+```python
+def test_ok():
+    assert True
+```
+"""):
+    tmp_md = str(tmpdir.join('notebook.md'))
+    with open(tmp_md, 'w') as fp:
+        fp.write(text)
+    jupytext([tmp_md, '--check', 'pytest'])
+
+
+def test_run_pytest_fail(
+        tmpdir,
+        capsys,
+        text="""This is a notebook with a test that should not pass
+```python
+def test_fail():
+    assert False
+```
+"""):
+    tmp_md = str(tmpdir.join('notebook.md'))
+    with open(tmp_md, 'w') as fp:
+        fp.write(text)
+    with pytest.raises(SystemExit):
+        jupytext([tmp_md, '--check', 'pytest'])