nbdev is all you need!!
By: The Mad Doctrine
Tags: Machine Learning, Data Science, Jupyter Notebook

nbdev offers end to end ML pipeline development capability in Jupyter notebooks!!

In Data Science/Machine learning projects there is always a split — between initial model training/exploration phase and final production development phase.

While most data scientist prefer working in Jupyter notebooks in initial exploration phase, they quickly move to other programming environments like PyCharm or VSCode as soon as code needs to be packaged for production environment.

What if I told you that there is a programming environment available where you never have to leave Jupyter notebooks and your end to end data science cycle can be managed from within notebooks itself.

nbdev offers just this capability & much more!!

Before nbdev came, I always felt the need for such a tool. For one having experienced IDE’s such as Visual Studio, I almost immediately fell in love with Jupyter notebooks as it offered a literate programming environment.

What is literate programming you would ask ?

Literate programming is a programming paradigm where code and documentation go hand in hand. (You can read more about literate programming here: Literate Programming.

The other advantage of Jupyter notebooks is that it’s assists in exploratory programming which is very useful in Data Science/Machine Learning as you can rapidly iterate and experiment with different algorithms and data exploration techniques.

However, as soon as the code was needed to be packaged in scripts, it became very difficult to work with, typically one had to copy paste all code into an IDE/editor and then start with version control, unit testing etc.

Enter nbdev created by Jeremy Howard, which offers all the below and more without leaving Jupyter notebook:

Automatic conversion of notebooks to python scripts, which are automatically bundled as python packages. Furthermore the python modules generated incorporate many best practices automatically.

merge and conflict resolution of notebooks in human readable format (which essentially means you can properly version control your notebooks ..❤

Define unit test from within notebooks, which are run automatically offering a Continuous Integration (CI) facility.

Automatic generation of code documentation website (which can also be integrated with Git pages). This is the feature which I absolutely .. ❤❤❤

What’s more if you are using GitHub, you get additional benefits such as:

Continuous integration on code push to GitHub via GitHub actions.

host your documentation using GitHub Pages.

If you are still not convinced see this video by Jeremy Howard:

Once nbdev has been installed and project created, the typical workflow consists of the below steps:

1. Use Jupyter notebooks to write your code, use appropriate ‘#’ special comments to specify which cells need to be exported in python modules, for example:

# default_exp <<module_name>>: to map notebook to corresponding python module.

# export : to export to python module



2. Once you have written the code notebooks, you can run “nbdev_build_lib” in command prompt to automatically generate corresponding python modules.

3. You can additionally run “nbdev_test_nbs” to run unit tests.

4. Finally you can run “make docs_serve” to generate documentation website for preview, something like this below:

While the installation instructions can be followed from the following link: [nbdev].

I found that the installation instructions are mainly keeping a Linux machine in mind.

Hence I am detailing out some of the steps below which are useful for successful installation on a Windows machine.

Thus this blog can serve as companion site in case you are having difficulties on running nbdev on Windows machine.