Repository: Project-MONAI/MONAI

Issue:
Move jupyter notebooks out of the project-monai/monai repo
**Background**
The `Project-MONAI/MONAI` repository now has [20 jupyter notebooks](https://github.com/Project-MONAI/MONAI/tree/master/examples/notebooks). They are mainly simple and intuitive working demos for the MONAI key features. With the upcoming MONAI educational events, MONAI will likely to have more demos in the form of jupyter notebooks.

**Proposal**
This ticket proposes to move the notebooks out of the `project-MONAI/monai` repo, to separate the maintenance efforts, and improve the usability. More specifically:

*Motivation*

- The notebooks require different types of maintenance efforts compared with the rest of the codebase.
it's currently unclear what/how we enforce the coding format, type hints, benchmarking, usability checks (see also https://github.com/Project-MONAI/MONAI/issues/872).
- Git version control is not very compatible with the notebooks,
this creates burdens for reviewing/verifying them (for example in pull request https://github.com/Project-MONAI/MONAI/pull/884, git diff is a pain)
- Some content requires additional packages such as matplotlib, it's unclear how these are managed and integration-tested. This should be decoupled from the main codebase.
- Some notebooks contain multimedia content which increases the main codebase size
- Currently, it's difficult to run notebooks with a pip-installed MONAI package. Because git-cloning notebooks will also automatically download the MONAI dev version (they are coupled). This potentially conflicts with the pip-installed monai at the system site-package level.

*A new home for the notebooks*

- option 1: create a Tutorial repo under `Project-MONAI` org (following the Pytorch org: https://github.com/pytorch/tutorials).
- option 2: convert the notebooks into html (with colab runnable links), deliver them via `http://monai.io/tutorials` pages
- option 3: open to any other suggestions to address the issues mentioned in the previous section


--------- summary of the discussions so far------------------
- can’t purely rely on colab (the internet access, python version, Numba issues)
- free plugin: ReviewNB which can help make the git diff of notebook changes
- notebooks and the core codebase have the tagging/versioning conflicts (need temp. workaround)
- notebooks tests could/should be automated
- key notebooks with the main repo, separate repo(s) for full tutorials
- notebooks need manual checks (e.g. visually check the figures)  before any milestones anyway


Discussion:
Thanks @wyli for your summary and for raising this discussion.
I want to add some concerns about moving the notebooks out of current repo:
1. China users **can't access Google** websites, including Colab, I think this is a big risk if we move all notebooks to Colab.
And even with VPN, I feel it's very slow to open Colab web pages.
2. Currently, all the notebooks have a button "run in Colab", if user clicks it, will open a Colab page to run this notebook directly, all the notebooks don't have external dependencies(Thanks for @benjamin-gorman 's help to add Colab support for all the notebooks). So we don't have any blocker to run the notebooks in Colab now.
3. I think the main problem here is how to maintain notebooks and review PRs about them. If we move all the notebooks to Colab directly, I don't know how users submit PR to modify notebooks or add new notebooks, if we move all the notebooks to a new repo, I still don't quite understand what's the benefits(reduce the size of core repo?).
4. To review jupyter notebook in Github, there is a free plugin: [ReviewNB](https://www.reviewnb.com/) which can help make the **diff more clear for review**.
5. Currently, all the notebooks will install the latest MONAI release version first if you run them, if users just cloned the MONAI source code and run it, it should not conflict with the dev version.

@ericspod , could you please also help share some comments?
Welcome a discussion about this topic.

Thanks.

@Nic-Ma good points, just wanted to clarify the fact that

there are external dependencies in the notebooks, which are not covered by `monai/requirements.txt` or `monai/requirements-dev.txt`.

Great to know that the notebooks use the pip installed monai release. The cloned core codebase is not actually needed/used when running the demos, which is another indicator that the notebooks shouldn't be part of the core codebase IMO.

just realised there's another issue of cyclic dependencies when releasing the package:
the notebooks require the tagging and releasing of the latest MONAI codebase, 
but the tagging and releasing depend on the readiness of all components of the codebase which currently includes the notebook folders.

So, If the notebooks use the latest tagging release, the notebook and the core codebase should be managed with different development lifecycles -- the core codebase should be released first.
Hi @wyli ,

> the notebooks require the tagging and releasing of the latest MONAI codebase

As we use `pip install monai` in notebooks now, notebooks don't have a dependency on explicit MONAI version.
I think we don't have this issue anymore?

Thanks.
There is an issue of dependency between the notebooks and the codebase, if we write a notebook to rely on the version of MONAI as committed with the notebook then that will be ahead of what we tag for pip. If what we choose to do is state that the notebooks should be for the most recent version installable via pip then they have to be explicitly written that way, which does exclude having experimental notebooks in the codebase. 

For versioning using a tools like like ReviewNB makes sense, converting to html isn't great because you lose the dynamic aspect and it sometimes doesn't get layed out as well. If we create a new repo for notebooks it has to be something people can check out and immediately use with some information on what dependencies exist such as matplotlib. 

We can't rely on Colab because of the access issue but also they're stuck on Python 3.6. The notebook I sent around about using Numba with Pytorch didn't work on Colab for example. I'm sure other things won't work either.


> Hi @wyli ,
> 
> > the notebooks require the tagging and releasing of the latest MONAI codebase
> 
> As we use `pip install monai` in notebooks now, notebooks don't have a dependency on explicit MONAI version.
> I think we don't have this issue anymore?
> 
> Thanks.

for a concrete example, https://github.com/Project-MONAI/MONAI/blob/master/examples/notebooks/automatic_mixed_precision.ipynb demonstrates the latest monai AMP capability, but `pip install monai` currently installs v0.2. v0.2 doesn't claim this feature. At least this causes confusions.
Hi @ericspod and @wyli ,

Thanks for your sharing. About the dependency issue, currently, if the notebook doesn't have a dependency after MONAI v0.2, we use `pip install monai` in the notebook directly, if it depends on new features, we temporarily use the `pip install -qU git+https://github.com/Project-MONAI/MONAI` and I will remove this line when we tag and release v0.3. For example:
https://github.com/Project-MONAI/MONAI/blob/master/examples/notebooks/mednist_GAN_workflow.ipynb

The AMP notebook is newly added, I am adding above logic as all the other new notebooks.

Thanks.
> Hi @ericspod and @wyli ,
> 
> Thanks for your sharing. About the dependency issue, currently, if the notebook doesn't have a dependency after MONAI v0.2, we use `pip install monai` in the notebook directly, if it depends on new features, we temporarily use the `pip install -qU git+https://github.com/Project-MONAI/MONAI` and I will remove this line when we tag and release v0.3. For example:
> https://github.com/Project-MONAI/MONAI/blob/master/examples/notebooks/mednist_GAN_workflow.ipynb
> 
> The AMP notebook is newly added, I am adding above logic as all the other new notebooks.
> 
> Thanks.

One of the main goals of this feature request is to remove this temporary workaround from the core codebase, now I'm confused...
Hi @wyli ,

1. No matter where we put the notebooks, I think we need to re-run all of them to double confirm before every release.
2. And these 3 new notebooks can only run with the latest code before v0.3 release, so must install MONAI from master code even put notebooks into other repo or Colab, right?

Anyway, I already updated the AMP notebook in PR: https://github.com/Project-MONAI/MONAI/pull/884.

Thanks.
Perhaps it would be good to keep two or three key notebooks (getting started, spleen segmentation, and one other) in the MONAI repo.   I think it is a great idea to start a (or many) separate repo(s) for tutorials and other notebooks.

Also, you can automate the testing of a jupyter notebook.   See the script:
https://github.com/InsightSoftwareConsortium/ITKTubeTK/blob/master/CMake/EvaluateIPythonNotebook.py




thanks for the discussions, based on the feedback, we plan to migrate the notebooks to a separate repo, and retain a readme file in this repo with links to them (will categorise the links into essential tutorials, latest features etc.)

notebook quality checks will be addressed by a separate ticket

Resolving patch:
diff --git a/examples/synthesis/gan_training.py b/examples/synthesis/gan_training.py
--- a/examples/synthesis/gan_training.py
+++ b/examples/synthesis/gan_training.py
@@ -17,7 +17,7 @@
     Extract tarball and set input_dir variable. GAN script trains using hand CT scan jpg images.
 
     Dataset information available in MedNIST Tutorial
-    https://github.com/Project-MONAI/MONAI/blob/master/examples/notebooks/mednist_tutorial.ipynb.
+    https://github.com/Project-MONAI/Tutorials/blob/master/mednist_tutorial.ipynb
 """
 
 import logging


Test patch:
diff --git a/examples/notebooks/multi_gpu_test.ipynb b/examples/notebooks/multi_gpu_test.ipynb
deleted file mode 100644
--- a/examples/notebooks/multi_gpu_test.ipynb
+++ /dev/null
@@ -1,236 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Multi GPU Test\n",
-    "\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Project-MONAI/MONAI/blob/master/examples/notebooks/multi_gpu_test.ipynb)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Setup environment"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "output_type": "stream",
-     "name": "stdout",
-     "text": "Note: you may need to restart the kernel to use updated packages.\n"
-    }
-   ],
-   "source": [
-    "%pip install -qU \"monai[ignite]\""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Setup imports"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [
-    {
-     "output_type": "stream",
-     "name": "stdout",
-     "text": "MONAI version: 0.2.0\nPython version: 3.7.5 (default, Nov  7 2019, 10:50:52)  [GCC 8.3.0]\nNumpy version: 1.19.1\nPytorch version: 1.6.0\n\nOptional dependencies:\nPytorch Ignite version: 0.3.0\nNibabel version: NOT INSTALLED or UNKNOWN VERSION.\nscikit-image version: NOT INSTALLED or UNKNOWN VERSION.\nPillow version: NOT INSTALLED or UNKNOWN VERSION.\nTensorboard version: NOT INSTALLED or UNKNOWN VERSION.\n\nFor details about installing the optional dependencies, please visit:\n    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies\n\n"
-    }
-   ],
-   "source": [
-    "# Copyright 2020 MONAI Consortium\n",
-    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
-    "# you may not use this file except in compliance with the License.\n",
-    "# You may obtain a copy of the License at\n",
-    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
-    "# Unless required by applicable law or agreed to in writing, software\n",
-    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
-    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
-    "# See the License for the specific language governing permissions and\n",
-    "# limitations under the License.\n",
-    "\n",
-    "import torch\n",
-    "\n",
-    "from monai.config import print_config\n",
-    "from monai.engines import create_multigpu_supervised_trainer\n",
-    "from monai.networks.nets import UNet\n",
-    "\n",
-    "print_config()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Test GPUs"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "lr = 1e-3\n",
-    "\n",
-    "net = UNet(\n",
-    "    dimensions=2,\n",
-    "    in_channels=1,\n",
-    "    out_channels=1,\n",
-    "    channels=(16, 32, 64, 128, 256),\n",
-    "    strides=(2, 2, 2, 2),\n",
-    "    num_res_units=2,\n",
-    ")\n",
-    "\n",
-    "\n",
-    "def fake_loss(y_pred, y):\n",
-    "    return (y_pred[0] + y).sum()\n",
-    "\n",
-    "\n",
-    "def fake_data_stream():\n",
-    "    while True:\n",
-    "        yield torch.rand((10, 1, 64, 64)), torch.rand((10, 1, 64, 64))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "### 1 GPU"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "source": [
-    "opt = torch.optim.Adam(net.parameters(), lr)\n",
-    "trainer = create_multigpu_supervised_trainer(net, opt, fake_loss, [torch.device(\"cuda:0\")])\n",
-    "trainer.run(fake_data_stream(), 2, 2)\n"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "execution_count": 4,
-   "outputs": [
-    {
-     "output_type": "execute_result",
-     "data": {
-      "text/plain": "State:\n\titeration: 4\n\tepoch: 2\n\tepoch_length: 2\n\tmax_epochs: 2\n\toutput: 40707.8984375\n\tbatch: <class 'tuple'>\n\tmetrics: <class 'dict'>\n\tdataloader: <class 'generator'>\n\tseed: 12"
-     },
-     "metadata": {},
-     "execution_count": 4
-    }
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "### all GPUs"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "source": [
-    "opt = torch.optim.Adam(net.parameters(), lr)\n",
-    "trainer = create_multigpu_supervised_trainer(net, opt, fake_loss, None)\n",
-    "trainer.run(fake_data_stream(), 2, 2)\n"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    },
-    "tags": []
-   },
-   "execution_count": 5,
-   "outputs": [
-    {
-     "output_type": "execute_result",
-     "data": {
-      "text/plain": "State:\n\titeration: 4\n\tepoch: 2\n\tepoch_length: 2\n\tmax_epochs: 2\n\toutput: 35669.37109375\n\tbatch: <class 'tuple'>\n\tmetrics: <class 'dict'>\n\tdataloader: <class 'generator'>\n\tseed: 12"
-     },
-     "metadata": {},
-     "execution_count": 5
-    }
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "source": [
-    "### CPU"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
-  },
-  {
-   "cell_type": "code",
-   "source": [
-    "opt = torch.optim.Adam(net.parameters(), lr)\n",
-    "trainer = create_multigpu_supervised_trainer(net, opt, fake_loss, [])\n",
-    "trainer.run(fake_data_stream(), 2, 2)"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "execution_count": 6,
-   "outputs": [
-    {
-     "output_type": "execute_result",
-     "data": {
-      "text/plain": "State:\n\titeration: 4\n\tepoch: 2\n\tepoch_length: 2\n\tmax_epochs: 2\n\toutput: 29662.359375\n\tbatch: <class 'tuple'>\n\tmetrics: <class 'dict'>\n\tdataloader: <class 'generator'>\n\tseed: 12"
-     },
-     "metadata": {},
-     "execution_count": 6
-    }
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}