Organise your Jupyter Notebook with these tips | by Zolzaya Luvsandorj | Towards Data Science

Data scientists do lots of exploration and experimentation. Jupyter Notebook (Notebook from here onwards) is a great tool for exploring and experimenting. However, things can get cluttered and messy quickly when using Notebook. Keeping your workflow clean, organised and easy-to-understand is an important skill and will serve you well in your professional career. In this post, I will share a few of my tips on how to keep your Notebook and workflow more organised.
Instead of putting your workflow from exploratory data analysis to modelling in one giant Notebook, it’s good to separate them out into a few logical pieces and name each Notebook descriptively. Let’s look at an example:
If we look at this example, we have a couple of Notebooks. From the name, we can see the sequence of Notebooks. For instance, we can guess that 3_base_models.ipynb precedes 4_gbm_model.ipynb while 4_gbm_model.ipynb and 4_rfc_model.ipynb are probably exploring two options in parallel. If we renamed these two to 4A_gbm_model.ipynb and 4B_rfc_model.ipynb, then we can hint an order in the name in a slightly different way as well.
You can have more or less number of Notebooks than the example. That is, the exact number of Notebook is not important. The key idea here is to have logical major tasks (it’s up to you how you define this) spread out, so it’s easier to see the overall workflow and find things. For instance, when writing production-ready code for deployment, we just have to refer to 5_final_pipeline.ipynb.
Markdown cells in Notebook allow us to be more expressive in how we document our workflow compared to code comments. We could use headers of different font size, ordered or unordered list of items, tables, mathematical formulas, checkboxes, hyperlinks, images and many more with Markdown. Here’s a one-minute introduction to Markdown on its commonly used syntax:
#### Headers# Header 1 ## Header 2#### Styles*Italic*, **bold**, _underscore_, ~~strikethrough~~#### Hyperlink[hyperlink](https://www.markdownguide.org/)#### Table| Default | Left-aligned | Center-aligned | Right-aligned ||---------|:-------------|:--------------:|--------------:|| Default | Left | Center | Right |#### Others* Bulleted item 1    * Bulleted subitem 1* Bulleted item 2***1. Ordered item 1      1.1. Ordered subitem 1  2. Ordered item 2***- [ ] Unchecked box- [x] Checked box
Here’s the rendered output of the syntax above:
Italic, bold, underscore, strikethrough
hyperlink
Bulleted item 1

Bulleted subitem 1


Bulleted subitem 1
Bulleted item 2

Ordered item 1
1.1. Ordered subitem 1
Ordered item 2

Ordered item 1
1.1. Ordered subitem 1
Ordered item 2
 Unchecked box
 Checked box
You can learn more about Markdown from here and here.
Pickles are not only delicious, but when used in the context of Python, they allow us to save objects as .pkl files. The process of saving the object is referred to as pickling.
One popular use of pickling in Data Science is saving trained models or machine learning pipelines into .pkl files. In the following example, we can see that there are couple of models saved in pickle:
This, for instance, will allow us to load the trained models or pipelines in different Notebooks for extended analysis of the predictions.
Another good use for pickles is to save objects like lists, dictionaries and the like. For instance, if we assigned manual selection of categorical features to a list in one Notebook and plan to also use it in another Notebook, we can write the list to a pickle file and load it in other Notebooks. This way if we manually make changes to the list, then we only have to do it in one place. Here’s an example code to write or load a pickle file named categorical.pkl where categorical refers to an object in Python (e.g. list).
If you want to learn more about pickles, check out this documentation.
You may have heard of the DRY principle: Don’t Repeat Yourself. If you haven’t heard of this software engineering principle before, it is about “not duplicating a piece of knowledge within a system”. One of my interpretation of this principle in Data Science is to create functions to abstract away the reoccurring tasks to reduce copy pasting. You can even use classes if it makes sense in your case.
Here are the suggested steps for this tip:
Create a functionEnsure the function has an intuitive nameDocument the function with docstring(Ideally) Unit test the functionSave the function in a .py file (.py file is referred as module)Import module in Notebook to access the functionUse the function in Notebook
Create a function
Ensure the function has an intuitive name
Document the function with docstring
(Ideally) Unit test the function
Save the function in a .py file (.py file is referred as module)
Import module in Notebook to access the function
Use the function in Notebook
Let’s try to contextualise these steps with examples:
Here’s a simple way to assess if a function has an intuitive name: If you think a colleague who hasn’t seen the function before could roughly guess what the function does just by looking at its name, then you are on the right track. When documenting these functions, I have adapted a few different styles in a way it made more sense to me. While these examples serve as a working example function for Data Science, I highly encourage you to check out official guides such as below to learn the best practices in naming and documentation conventions, style guides and type hints:
PEP 8 for style guide
PEP257 for docstring conventions
PEP484 for type hints
Example Numpy Style Python Docstring
Example Google Style Python Docstrings
You can even browse through modules in well-established package’s Github repository to get inspirations.
If you saved these functions in a helpers.py file and imported helpers module (by the way, Python module just means a .py file) in our Notebook with import helpers, you can access the documentation by writing the function name followed by Shift + Tab:
If you have many functions, you could even categorise and put them in separate modules. If you take this approach, you may even want to create a folder containing all the modules. While putting stable code into a module makes sense, I think it is fine to keep experimental functions in your Notebook.
If you implement this tip, you will soon notice that your Notebook start to look less cluttered and more organised. In addition, using functions will make you less prone to silly copy paste mistakes.
Unit testing was not covered in this post as it deserves its own section. If you would like learn about unit testing for Data Science, this PyData talk may be a good starting point.
It’s not uncommon to be tweaking parts of your code anywhere in your Notebook when experimenting. However, once you are done experimenting, clean it up, restart the kernel and ensure cells in your Notebook ran top-down once such that you won’t have out of order cells like this:
Voila❕ These were my tips for organising your Jupyter Notebook workflow.✨
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my post. I hope you will get to use these tips to have a more organised workflow. If you are interested, here are links to some of my posts:
◼️ Simple data visualisations in Python that you will find useful◼️ 6 simple tips for prettier and customised plots in Seaborn (Python)◼️ Exploratory text analysis in Python◼️️ 5 tips for pandas users◼️️ 5 tips for data aggregation in pandas◼️️ Writing 5 common SQL queries in pandas◼️️ Writing advanced SQL queries in pandas
Bye for now 🏃💨