Stop copy-pasting notebooks, embrace Jupyter templates! | by Emanuele Fabbiani | Towards Data Science

Jupyter notebooks are awesome. They combine code, output and text, enabling fast prototyping, early debugging and effective storytelling.
Nonetheless, notebooks come with their own limitations, the most annoying ones being limited scalability, tendency to push developers to duplicate code and lack of a standard structure.
Lots of articles, posts and talks deal with these topics. Two of them have special relevance for the sake of this article. Joel Grus, in I don’t like notebooks, accuses notebooks for instillating bad habits in developers and data scientist. Answering Joel on Medium, Will Koehrsen suggest to blame sins and not sinners and proposes an interesting solution.
A template. A standard yet flexible skeleton for every kind of Data Science notebook. If people are provided with a proper structure to fill with their content, Will argues, they are nudged to implement best practices.
The benefit of this approach is that it changes the defaults.
When we read Will’s post, xtream’s data science team and I immediately looked for a package implementing the solution. Unfortunately, we found that no such thing existed on PiPy or conda.
Then, we decided to build our own: we developed a Jupyter extension with two main features.
First, it allows you to create every new notebook from a template. The template is a notebook itself, so that it can be easily edited by users and adapted to different needs. We also provide the possibility of inserting the template in an existing notebook.
The default template, shipped with the extension, is made of 6 sections:
Abstract: a quick summary of the notebook, is terms of its purpose, methodology, results and suggested next stepsSetup: package and local module imports, with the most common data science libraries already imported with their standard aliasParameters: definition of the parameters which affect the notebookData import: statements to load all the data to be used in the notebookData processing: core processing, often to be further divided into sub-sectionsReferences: useful references to literature or other notebooks
Abstract: a quick summary of the notebook, is terms of its purpose, methodology, results and suggested next steps
Setup: package and local module imports, with the most common data science libraries already imported with their standard alias
Parameters: definition of the parameters which affect the notebook
Data import: statements to load all the data to be used in the notebook
Data processing: core processing, often to be further divided into sub-sections
References: useful references to literature or other notebooks
It is important to note that this is just a default template. Every Data Scientist may create its own. If you happen to do so, let me know so that we may improve our repository.
Second, every time a notebook is saved as Untitled, the user is asked to rename it.
Of course, both the features can be enabled or disabled from the configuration of the extension.
Notable improvements became visible as soon as we introduced the usage of the extension in our workflow.
With a common skeleton, notebooks from other team members became easier to read and share. The Abstract, in particular, provided us with a 2-minute summary, allowing the reader to decide whether to dive deep into the code or look into other places.
With sections ready to be filled, we started feeling guilty when documenting our code superficially. This led to longer and more insightful comments, thus improving the overall readability of the notebooks.
With local imports configured and ready to be used, there was no excuse to prevent refactoring of relevant code out of the notebook and into common modules.
With naming conventions explicit in the template, no one could forget about them.
Finally, with standard imports embedded in the template, we were able to save some minutes googling for the way to correctly import modules.
Our extension is available in the package jupytemplate, open-source and published on GitHub: https://github.com/xtreamsrl/jupytemplate.
Although not strictly required, we suggest to start downloading the whole set of jupyter extensions, which currently does not include jupytemplate.
Then, install jupytemplate by running:
pip install jupytemplate
Finally, enable the extension in your local Jupyter instance:
jupyter nbextension install --py jupytemplate --sys-prefixjupyter nbextension enable jupytemplate/main --sys-prefix
More details in the documentation on GitHub.
That’s it. You should now be able to do something like this.
Thank you, my reader, for getting here!
Every comment, question, or general feedback is always welcome. You can get in touch with me on LinkedIn (link below).
If you appreciated this article, you may be interested in:
Lessons from a real Machine Learning project, part 1: from Jupyter to Luigi
Lessons from a real Machine Learning project, part 2: traps of Data Exploration
Introducing tsviz, interactive time series visualization in R Studio
If you are curious about about me or xtream, check us out on LinkedIn!