Automate the Structure of Your Data Science Projects with Cookiecutter | by Ahmed Besbes | Towards Data Science

Here’s a situation a lot of data scientists are familiar with.
Every time you start a new project, you reuse the structure of older projects. You go through their folders and copy-paste them, you remove the unnecessary files, you rename the remaining ones with your new project’s specifics, and you go inside each configuration file and replace the old environment variables (URLs, API keys, Hosts, Ports, etc.) with the new ones.
And maybe, who knows, you’ll create new configurations and new folders along the way.
❌ We all agree that this is a pretty tedious and repetitive task. Not to mention that it’s prone to errors.
✅ Wouldn’t be more convenient to start each new project from a master template that you’d clone and fill in with the specific information from the terminal? Wouldn’t be great to let this template automatically build a whole folder structure for you, and populate the files with the right names and variables you define?
Cookiecutter is the right solution to this problem.
In this post, we’ll have a look at cookiecutter.We’ll understand how it works and how you can use it to build custom and reusable templates for your projects.Then, we’ll cover the Cookiecutter Data Science open-source template to kickstart data science projects that follow the best standards in the industry.
Without further ado, let’s have a look 🔍
“Cookiecutter creates projects from project templates.” — official doc
Projects can be python packages, web applications, machine learning apps with complex workflows or anything you can think of
Templates are what cookiecutter uses to create projects. They rely on Jinja2’s syntax
What cookiecutter does is quite simple: it clones a directory and put it inside your new project. It then replaces all the names that are between {{ and }} (Jinja2 syntax) with names that it finds in the cookiecutter.json file. (We’ll see an example of how to build a cookiecutter template in the next section)
To get started using cookiecutter, you can install it with pip:
pip install cookiecutter
or conda:
conda install -c conda-forge cookiecutter
👉 You can use cookiecutter on all platforms: Windows, Mac and Linux👉 It works with Python 2.7+ and 3.5+(although prefer Python 3.5+ since Python 2.7 is no longer maintained)👉 You can use Cookiecutter to create templates in one or multiple languages
In this section, I’ll show you how to create a cookiecutter template to kickstart Streamlit projects.
If you’re not familiar with Streamlit, it’s a Python library designed to build web applications. It’s very simple to use and provides a lot of functionalities. I use it on a daily basis to share experiments and results with my team and to prototype machine learning apps.
My Streamlit-based projects tend to have the following structure:
An src folder that contains the main script of the app (app.py ) as well as a utils module that contains two scripts: ui.py to put the layout functions and common.py to hold other utility functions for data processing or remote database connections (among other things)
A .gitignore file to prevent git from versioning unnecessary files (such as .env files, or .pyc files)
Procfile and setup.sh : to handle the deployment on Heroku
requirements.txt : to list the project dependencies
A .env file to store the environment variables of the project
A README.md to share details about the project
To create a cookiecutter template that generates this structure, let’s start by creating a folder for this template.
mkdir streamlit-cookiecuttercd streamlit-cookiecutter
Inside this folder, create the cookiecutter.json file:
A few things about this file:
Each key is an item that cookiecutter will ask for when it kickstarts a project from the template
Each value corresponds to the default value of each item
The values of the items are set in order. For example, the repo_name is set after the project_name (since its value depends on project_name ‘s value)
Setting the values of these items happens when you clone the template with cookiecutter: we’ll see how this is done in the next step.
Now create the folder and put the desired target structure in it. We’ll name this folder based on the repo_name item. In cookiecutter syntax: {{cookiecutter.repo_name}} .
Inside the {{cookiecutter.repo_name}} folder, put the desired structure that you want into your projects:
Each of these files can access the values of the items you pass to cookie-cutter: all you have to do is use {{ and }} .
Let’s see how we can exploit this.
👉 Generate a README.mdWe can automatically generate the README.md by inserting the items project_name , description , open_source_license in it.
{{cookiecutter.description}}
{{cookiecutter.open_source_license}}
👉 Set environment variablesWe can automatically fill in the values of s3_bucket , aws_profile,port , host and api_key inside the .env file.
👉 Deduplicate codeIf your Streamlit apps follow the same structure and all start with the name of the project as the title, there’s no need to repeat this code every time.
Once you’re done with your template’s code, push it to Github. I’ve done it and you can check it here.
Now that the template is on Github, let’s use it to start a project.
Let’s say that I want to create a sentiment analysis app in Streamlit.
All I need to do is call cookiecutter with the URL of the template.
cookiecutter git@github.com:ahmedbesbes/streamlit-cookiecutter.git
Once you execute this command, Cookiecutter will ask you to set the values of the items you defined in the cookiecutter.json file (notice that the default value of each item is put between brackets).
After entering the specific values for each item, the project is created.
Let’s check its README.md✅
Its .envfile ✅
Its app.pyfile ✅
All good. The project has the desired structure and the files are populated with the right data.
Now we can start working on this app.
One great thing about cookiecutter is the vibrant community. There’s a bunch of open-source templates out there and of different flavours (Django, Flask, FastAPI, you name it). You can easily look them up on Github and start using them.
One particular template that caught my attention is Cookiecutter Data Science.
As the authors say, this template is:
A logical, reasonably standardized, but flexible project structure for doing and sharing data science work.
What this template provides in practice, is a set of directories to better organize your work. It’s slightly opinionated, but it follows good practices that the field agrees on.
In my opinion, it helps:
Organizing the modules by responsibility: dataset creation, feature engineering, modelling, visualization
Organizing the data by pipeline stage
Enforcing unit tests
Creating documentation
You can start using this template like the following:
Once you’re done setting the values for each item, you’ll get the following structure:
You don’t have to stick with this structure: if you’re not happy with it, you can fork the template and modify it to fit your needs.
Cookiecutter is a fantastic library. It helps automate project creation and prevents you from repeating yourself. As a data scientist, it must be part of your toolbox.
To learn more about Cookiecutter, I’ve selected some good resources you can easily go through:
https://github.com/cookiecutter/cookiecutter
https://drivendata.github.io/cookiecutter-data-science/
https://dev.to/azure/10-top-tips-for-reproducible-machine-learning-36g0
https://towardsdatascience.com/template-your-data-science-projects-with-cookiecutter-754d3c584d13
https://youtu.be/nExL0SgKsDY (an excellent Youtube video)
If you’ve stuck till the end, I really thank you for your time and hope that you learned something about cookiecutter and project templating.
That’ll be all for me today. Until next time for more programming tips and tutorials. 👋