Version Control for Jupyter Notebook
By: Ceshine Lee
Tags: Python, Data Science, Jupyter Notebook

Jupyter Notebook Best Practices for Data Science by Jonathan Whitmore has been incredibly helpful and I’ve been strongly encouraging team members to adopt at least a subset of them, specifically the post-save hook and the notebook naming convention.

Post-save Hook

The idea is to automatically save a copy of the notebook in both Python script and HTML format whenever the notebook is saved.

I used to clear all output in the notebook before committing it to the version control to make change history cleaner, but that creates two problems:

If the data processing part of the notebook requires more than ten minutes to run, it would create a strong incentive for the reviewer to skip actually running the notebook, and instead just read through the code and assuming it does what it appears to do. This is assuming the trained model is serialized in some way for the reviewer to load, otherwise it may take hours to train on reviewer’s system. The distribution of the pre-trained model brings up the next problem: There’s no easy way to distributed the data and model files along with the notebook. Storing and transferring gigabytes of data isn’t that easy for data scientist, and asking data engineers to build something internally just for this may not be cost-effective. For “Big Data” scenario, you wouldn’t want to save Spark access credentials inside the notebooks (really dangerous), so you’d need to find some way to securely manage the cluster access. (There’s a new framework called Data Version Control that looks promising. But you’d need to pay fees to Google or Amazon for the cloud storage.)

All in all, the most direct way to solve these problems is to just include the outputs of the notebook in the version control. (Though I’d recommend restart the kernel and re-run the whole notebook sequentially before committing.)

Convert the notebook to a Python script and a HTML file. Use the Python script to do code diff in the review process. HTML file works best as an attachment in cross-team communication emails. (Personally I’d only add HTML files to version control only if there are active participants of the project who don’t really use Jupyter.)

The following gist is basically the direct copy from Jupyter Notebook Best Practices for Data Science. It’s tested against the latest Jupyter and put here for quick reference:

There are two ways to use the above script:

Put it at the top of ~/.jupyter/jupyter_notebook_config.py . (Run jupyter notebook --generate-config to create it if you don’t have it yet) Use export JUPYTER_CONFIG_DIR=~/.jupyter_profile2 to specify the config folder if you want different configurations for different projects. Put it in the root of the project folder (where you run jupyter notebook command) as jupyter_notebook_config.py .

There’s more!

Jupyter notebook, if used properly, can be a very high leverage tool that boosts the efficiency of team collaboration. Please check out Jupyter Notebook Best Practices for Data Science for more.

20190313 Update

There’s a new tool that enables direct Diff of Jupyter notebook on Github:

I haven’t found time to really try it yet, but I’ve heard good things about it: