Repository: pykale/pykale (language: python)

Issue:
[Feature Request] Tests for notebooks
# 🚀 Feature Request

Add basic tests for the tutorial notebooks. 

## Motivation

Changes to dependencies can cause notebooks to fail unexpectedly.

## Pitch

Using GitHub actions, these should:

- Set up an appropriate execution environment
- Convert the notebook to a script
- Run the script
- Report failure if there is an error

**Describe alternatives you've considered**

Regression tests would be very time consuming.

**Are you willing to open a pull request?** Yes



Discussion / hints:
@bobturneruk Will this be implemented soon? We got another problem with the recent PR #218, where the bug in the digits notebook is undetected. I only manually discovered today.
https://colab.research.google.com/github/pykale/pykale/blob/main/examples/digits_dann_lightn/tutorial.ipynb

This could be embarrassing because we are using notebooks as the primary starting points and we are now publicising it.

Sorry to hear about that. I will make some time for this tomorrow.
https://github.com/marketplace/actions/nbmake

Fix patch:
diff --git a/examples/bindingdb_deepdta/tutorial.ipynb b/examples/bindingdb_deepdta/tutorial.ipynb
index 285f3e9b5..a94881b76 100644
--- a/examples/bindingdb_deepdta/tutorial.ipynb
+++ b/examples/bindingdb_deepdta/tutorial.ipynb
@@ -329,6 +329,9 @@
    "language": "python",
    "name": "python3"
   },
+  "execution": {
+      "timeout": 600
+  },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
diff --git a/examples/cmri_mpca/tutorial.ipynb b/examples/cmri_mpca/tutorial.ipynb
index 3806bd0d5..baa1d5394 100644
--- a/examples/cmri_mpca/tutorial.ipynb
+++ b/examples/cmri_mpca/tutorial.ipynb
@@ -375,6 +375,9 @@
    "language": "python",
    "name": "python3"
   },
+  "execution": {
+      "timeout": 600
+  },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
diff --git a/examples/digits_dann_lightn/tutorial.ipynb b/examples/digits_dann_lightn/tutorial.ipynb
index 91717abe0..c3ac805d4 100644
--- a/examples/digits_dann_lightn/tutorial.ipynb
+++ b/examples/digits_dann_lightn/tutorial.ipynb
@@ -360,6 +360,9 @@
    "language": "python",
    "name": "python3"
   },
+  "execution": {
+      "timeout": 600
+  },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
diff --git a/setup.py b/setup.py
index 3f59b0ef2..28470b23f 100644
--- a/setup.py
+++ b/setup.py
@@ -37,9 +37,11 @@
     "coverage",
     "flake8",
     "flake8-print",
+    "ipywidgets",
     "isort",
     "m2r",
     "mypy",
+    "nbmake>=0.8",
     "nbsphinx",
     "nbsphinx-link",
     "nbval",


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 28007e8f8..b8597161a 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -46,7 +46,7 @@ jobs:
         shell: bash -l {0}
       - name: Run unit tests
         run: |
-          pytest --cov=kale
+          pytest --nbmake --cov=kale
         shell: bash -l {0}
       - name: Determine coverage
         run: |