Repository: kubeflow/pipelines

Issue:
Pipeline component for papermill?
Would it be useful to have a pipeline component that could run a notebook using papermill?
https://github.com/nteract/papermill

papermill is a system for executing notebooks from Netflix.


Discussion:
/cc @aronchick 
We already have a notebook sample test using the papermill to parameterize the notebooks.
https://github.com/kubeflow/pipelines/blob/ad1950b8e63961a56fb8e0ad1efd45a8a6330b37/test/sample-test/run_test.sh#L282
I just merged the PR.
I think there's a real opportunity to do some super cool things here - for example - 
https://github.com/nteract/papermill

- Make the default notebook in Jupyter "parameterizable" via configMap (or other) namespace variables
- Collect all statistics in a PV for sharing (don't we also have this problem with TensorBoard/etc)
- Executing notebooks (particularly non-distributed ones) using native Kubernetes constructs

Am I talking crazy?
I like the idea of launchers.
We have some code in form of command line or container task or notebook or source-code URL.
We can launch that code using different launchers: directly on single node, on CMLE, on Kubeflow etc.
+1

This could be used to generate visualization based on the execution of a notebook. 
We are currently using Airflow (utilizing solely the`KubernetesPodOperator`) to launch jobs with great success. Notebook jobs (Docker+papermill)  plays a big role. Now that you guys, support pipelines neatly. I'm actually thinking that we should switch to using kubeflow instead.

One of the great wins is that we get to look at the outputs of the runs by storing the notebooks in an s3 bucket which https://github.com/nteract/commuter is linked to. In Data Science, this is specifically useful since we can look at the actual plots as well and compare results for debugging etc. It would be nice if this would be improved or even integrated with kubeflow. Comparing plots/notebooks between runs etc. 

I think it would be neat if you could via the DAG graph visualisation of a pipeline that has run, get linked to a rendering of the generated output notebook. 

In general, I think it we will end up in a similar situation with kubeflow. We have multiple components wired towards specific tasks and  programming languages. Same goes for specifying profiles for our jupyterhub instance. 

I would like to help out on this project. 
This is so cool! Is there anything we can do to help here?
This dude is seems to be on to something. I think it would make sense to help him out a little https://kubeflow-kale.github.io/

eg. provide an official jupyterlab extension for kubeflow. where you could deploy notebook as a pipeline (papermill). Furthermore, having individual cells as lightweight components seems like a pretty good idea. 

What we ended up doing was using shorter notebooks deployed with Airflow kubernetesPodOperator & papermill. The work that he has started would be more practical. 

Anyways, it seems like a good alternative option to the SDK 
@LeonardAukea I think it would be easy for me to introduce this feature.
I've already added support for Airflow operators (via [create_component_from_airflow_op](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.create_component_from_airflow_op)) and running notebooks is much easier.

However we first need to solve the following design problem:
How do you specify inputs and outputs of the notebook? Components correspond to functions or entrypoints, but notebook is a script - it does not have natural parameters or return values.

What would be the way to add specify inputs and outputs in the notebook?
How should we support running notebooks that were created before this feature?
@Ark-kun, I was thinking about this desing. The standard parameters for a papermill execution are the input and output notebooks and the parameters dictionary.

How reasonable is to expect a general output file? Such as a json or a yaml. 

I think it is already expected that some other outputs, such as data and models, to be stored in an external persistence. Maybe the correct step is to make it explicit in the documentation.

In this case it will not be ideal but it could  be used as a base component that can be detailed later by the user.

Also, I think that it would be nice to add a flag for the parameters cell, indicating to accept or not to execute a notebook without an explicit parameters cell.


@otaviocv Can you write a small mockup? How does the notebook look and what arguments do you pass to the papermill component.
Yes, I can.
@Ark-kun, I have tried to develop a simple mockup but didn't time to check integrations with external file systems. My mockup is [here](https://github.com/otaviocv/pipelines/tree/papermill-mockup/components/local/papermill).

Papermill has native io with external file systems, like [AWS](https://papermill.readthedocs.io/en/latest/reference/papermill-io.html#papermill.iorw.S3Handler) and [GCS](https://papermill.readthedocs.io/en/latest/reference/papermill-io.html#papermill.iorw.GCSHandler), for notebook read and write, even [Http](https://papermill.readthedocs.io/en/latest/reference/papermill-io.html#papermill.iorw.HttpHandler).

My idea is that the component would have three inputs:
- input notebook
- output notebook
- parameters dictionary (a yaml file)

All these files stored in an external file system or maybe the input notebook and the parameters yaml from a raw path to a git repository.

And one output, maybe a json or yaml with more complex information, or just a text file with some simple information. This would serve as a general papermill component and could be modified just on the output without having to change the implementation or the docker image, just the component description to match the particular output.

What do you think?

I would love to tackle this feature if it is possible... ;)
@otaviocv Looks like the mockup component does not seem to have any outputs that could be passed forward. But even if it passed out the completed notebook, would it be useful for any downstream tasks? Would some component be able to consume a completed notebook?
Did this end up going anywhere?  Seems like a nice feature to have
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

/freeze

Resolving patch:
diff --git a/components/notebooks/samples/sample_pipeline.py b/components/notebooks/samples/sample_pipeline.py
new file mode 100644
--- /dev/null
+++ b/components/notebooks/samples/sample_pipeline.py
@@ -0,0 +1,20 @@
+kfp_endpoint = None
+
+import kfp
+from kfp import components
+
+download_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/240543e483076ae718f82c6f280441daa2f041fd/components/web/Download/component.yaml')
+run_notebook_op = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/4ebce5f643b6af5639053ea7eaed52b02bf7e928/components/notebooks/Run_notebook_using_papermill/component.yaml')
+
+def notebook_pipeline():
+    notebook = download_op('https://raw.githubusercontent.com/kubeflow/pipelines/93fc34474bf989998cf19445149aca2847eee763/components/notebooks/samples/test_notebook.ipynb').output    
+    
+    run_notebook_op(
+        notebook=notebook,
+        parameters={'param1': 'value 1'},
+        input_data="Optional. Pass output of any component here. Can be a directory.",
+        packages_to_install=["matplotlib"],
+    )
+
+if __name__ == '__main__':
+    pipelin_run = kfp.Client(host=kfp_endpoint).create_run_from_pipeline_func(notebook_pipeline, arguments={})


Test patch:
diff --git a/components/notebooks/samples/test_notebook.ipynb b/components/notebooks/samples/test_notebook.ipynb
new file mode 100644
--- /dev/null
+++ b/components/notebooks/samples/test_notebook.ipynb
@@ -0,0 +1,82 @@
+{
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2,
+ "cells": [
+  {
+   "source": [
+    "# Parameters\n",
+    "INPUT_DATA_PATH = INPUT_DATA_PATH or \"\"\n",
+    "OUTPUT_DATA_PATH = OUTPUT_DATA_PATH or \"\""
+   ],
+   "cell_type": "code",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "# Show the parameter values\n",
+    "print('INPUT_DATA_PATH = ' + INPUT_DATA_PATH)\n",
+    "print('OUTPUT_DATA_PATH = ' + OUTPUT_DATA_PATH)\n",
+    "print('locals() = ' + str(locals()))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Checking the input data\n",
+    "import os\n",
+    "\n",
+    "if INPUT_DATA_PATH:\n",
+    "    if os.path.isdir(INPUT_DATA_PATH):\n",
+    "        print('os.listdir(INPUT_DATA_PATH):')\n",
+    "        print(os.listdir(INPUT_DATA_PATH))\n",
+    "    if os.path.isfile(INPUT_DATA_PATH):\n",
+    "        print('os.stat(INPUT_DATA_PATH):')\n",
+    "        print(os.stat(INPUT_DATA_PATH))\n",
+    "else:\n",
+    "    print('INPUT_DATA_PATH is empty')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Writing some output data\n",
+    "from pathlib import Path\n",
+    "\n",
+    "(Path(OUTPUT_DATA_PATH) / 'output.txt').write_text(\"Hello world!\")"
+   ]
+  }
+ ]
+}
\ No newline at end of file