Repository: ploomber/soorgeon (language: python)

Issue:
A utility to ensure the notebook runs
Before refactoring a notebook, the user must ensure that the original notebook runs. We should have a command to check whether it works and suggest actions if if there are some errors.

e.g., 

```sh
soorgeon run nb.ipynb
```

If the notebook fails because of a `ModuleNotFound`: suggest creating a virtualenv, and adding a `requirements.txt` with the package name

If it's an other error, show the guide for debugging notebooks

if it's a function signature, recommend downgrading some libraries

An alternative approach would be to let the user convert anyway and then help them fix the issues in the ploomber generated pipeline, this way they can leverage incremental builds for rapid iterations. we can also suggest them to add a sample parameter



Discussion / hints:
@edublancas currently it just throws an error while refactoring right?
I'm thinking we can put it as an independent command + coupled by default as part of the refactor, thoughts?
No, it doesn't throw an error. The refactoring command would pass but the code won't run.

I think this could be semi-automated: run the notebook and try to fix as many problems as we can without user intervention. The problem is that it isn't simple to implement an efficient solution. For example, imagine that the notebook doesn't have a `requirements.txt`, and it breaks on each `import package`; we could capture the import errors and add the package names to `requirements.txt` but then we'd have to restart the execution of the notebook over and over again.

I think we should leave this off right now as it isn't simple to fix.

As a first step, we need a command called test, to run the notebook prior to refactoring and making sure it can run without errors. Then we can have multiple options there, like matching before/after refactoring etc.
For notebooks, we can use either `papermill {test_file_name.ipynb} {test_file_output.ipynb}` (or library function`papermill.execute_notebook`) or `jupyter nbconvert --to notebook --execute mynotebook.ipynb` and  then check if the output contains any exception.

For python files, we can use `exec(open("filename.py").read())`.

----

> If it's an other error, show the guide for debugging notebooks

Should we just put a link that points to https://docs.ploomber.io/en/latest/user-guide/debugging.html?

> if it's a function signature, recommend downgrading some libraries

Do you mean `AttributeError`? For example when I run
```
import math
math.func()
```
It gives the error `AttributeError: module 'math' has no attribute 'func'.`
The way we do it currently with the `ploomber package` is using nbconvert from ipynb to .py and then execute it, I think we should keep the same convention here (at least for now, for simlicity). @edublancas thoughts?

> Should we just put a link that points to https://docs.ploomber.io/en/latest/user-guide/debugging.html?

Yes + probably show the error code.

> Do you mean AttributeError? For example when I run

I think that what he means, usually on dependency discrepancies, there are missing functions, arguments, or it expects an extra member, AttributeError is probably a part of this set. @edublancas is that right??
A few things:

On second thought, I think we should name this `soorgeon test`. The objective of this command is to help anyone fix their notebooks until they ensure they work properly. We'll start adding a bunch of features to this command but for the moment being, let's start with the basics of running the notebook.

To execute the notebook, let's use Ploomber. It already wraps papermill and supports .py files. Essentially, we'd wrap the notebook into a pipeline of a single task. [The documentation has an example](https://docs.ploomber.io/en/latest/api/_modules/tasks/ploomber.tasks.NotebookRunner.html#ploomber-tasks-notebookrunner) of how to do it with the Python API. This would cause soorgeon to have ploomber as a dependency but I think that's fine.


> I think that what he means, usually on dependency discrepancies, there are missing functions, arguments, or it expects an extra member, AttributeError is probably a part of this set. @edublancas is that right??

Yes, attribute error is only one type of error. Another can be passing an argument that no longer exists. I'd say let's start with the executing logic and then we think of this error handling thing.


Can we run `ploomber.tasks.NotebookRunner` before refactoring the notebook?

```
from pathlib import Path
from ploomber import DAG
from ploomber.tasks import NotebookRunner
from ploomber.products import File
dag = DAG()
NotebookRunner(Path('nb.ipynb'), product=File('report.html'), dag=dag)
```
When I tried to run this on `soorgeon/examples/exploratory/nb.ipynb`, it will output that it requires a `tags=["parameters"]` section. 

Looks like it only works for refactored notebook. Not sure if my understanding is correct.
Maybe try using [papermill](https://papermill.readthedocs.io/en/latest/usage-execute.html#execute-via-the-python-api) to avoid circular dependency?
Unless we can utilize some of the other code in `NotebookRunner` (it doesn't seem we need anything besides executing it at the moment).
> When I tried to run this on soorgeon/examples/exploratory/nb.ipynb, it will output that it requires a tags=["parameters"] section.

good point. i forgot that ploomber requires the parameters cell.

so yeah, let's go ahead and use papermill. you'd need to add some extra logic if the input is a .py file, you can use jupytext to convert it into ipynb https://github.com/mwouts/jupytext

please also add both papermill and jupytext to the dependencies in setup.py





Fix patch:
diff --git a/README.md b/README.md
index f4319ac..e938afe 100644
--- a/README.md
+++ b/README.md
@@ -42,6 +42,20 @@ pip install soorgeon
 
 ## Usage
 
+### Testing if the notebook runs (Optional)
+
+Before refactoring, you can optionally test if the original notebook or script runs without exceptions:
+
+```
+soorgeon test path/to/notebook.ipynb
+```
+
+or
+
+```
+soorgeon test path/to/script.py
+```
+
 ### Refactoring
 
 ```sh
@@ -78,7 +92,7 @@ soorgeon clean path/to/notebook.ipynb
 or
 
 ```
-soorgeon clean path/to/script.ipynb
+soorgeon clean path/to/script.py
 ```
 
 ## Examples
diff --git a/src/soorgeon/cli.py b/src/soorgeon/cli.py
index 20f60cb..dee2d9f 100644
--- a/src/soorgeon/cli.py
+++ b/src/soorgeon/cli.py
@@ -1,4 +1,7 @@
 import click
+import tempfile
+import jupytext
+from os.path import abspath, dirname
 from soorgeon import __version__, export
 from soorgeon.clean import basic_clean
 
@@ -96,3 +99,54 @@ def clean(filename):
 
     """
     basic_clean(filename)
+
+
+@cli.command()
+@click.argument("filename", type=click.Path(exists=True))
+def test(filename):
+    """
+    check if a .py or .ipynb file runs.
+
+    $ soorgeon test path/to/script.py
+    or
+    $ soorgeon test path/to/notebook.ipynb
+
+    """
+    if filename.lower().endswith(".ipynb"):
+        nb = jupytext.read(filename)
+        # convert ipynb to py and create a temp file in current directory
+        directory = dirname(abspath(filename))
+        with tempfile.NamedTemporaryFile(suffix=".py",
+                                                delete=True,
+                                                dir=directory) as temp_file:
+            jupytext.write(nb, temp_file.name)
+            _test(temp_file.name)
+    else:
+        _test(filename)
+
+
+def _test(filename):
+    try:
+        exec(open(filename).read())
+        click.echo(f"Finished executing {filename}, no error encountered")
+    except (ModuleNotFoundError, AttributeError, SyntaxError) as err:
+        error_suggestion_dict = {
+            "ModuleNotFoundError": "create a virtualenv, and"
+            " adding a requirements.txt with the package",
+            "AttributeError": "downgrade some libraries",
+            "SyntaxError": "check syntax",
+        }
+        error_type = type(err).__name__
+        click.echo(f"""
+        {error_type} encountered while executing the notebook: {err}
+
+        It is recommended to {error_suggestion_dict[error_type]}
+        """)
+    except Exception as err:
+        error_type = type(err).__name__
+        click.echo(f"""
+        {error_type} encountered while executing the notebook: {err}
+
+        Checkout how to debug notebooks:
+        https://docs.ploomber.io/en/latest/user-guide/debugging.html
+        """)


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 93b0afe..0510789 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -501,3 +501,52 @@ def test_refactor_product_should_not_warning_if_comment(tmp_empty):
     assert 'to_parquet' not in result.output
     assert 'write_text' in result.output
     assert 'write_bytes' not in result.output
+
+
+ModuleNotFoundError_sample = """
+# ## header
+
+import nomodule
+# ## header
+
+"""
+
+AttributeError_sample = """
+# ## header
+
+import math
+print(math.logg(1))
+# ## header
+"""
+
+SyntaxError_sample = """
+# ## header
+
+impor math
+print(math.log(1))
+# ## header
+"""
+
+OtherError_sample = """
+# ## header
+
+import math
+print(math.log(-5))
+# ## header
+"""
+
+
+@pytest.mark.parametrize('code, output', [
+    [simple, "no error encountered"],
+    [ModuleNotFoundError_sample, "It is recommended to create a virtualenv"],
+    [AttributeError_sample, "It is recommended to downgrade some libraries"],
+    [SyntaxError_sample, "It is recommended to check syntax"],
+    [OtherError_sample, "Checkout how to debug notebooks"]
+])
+def test_test_notebook_runs(tmp_empty, code, output):
+    nb_ = jupytext.reads(code, fmt='py:light')
+    for filename in ['nb.ipynb', 'nb.py']:
+        jupytext.write(nb_, filename)
+        runner = CliRunner()
+        result = runner.invoke(cli.test, filename)
+        assert output in result.output