Cookiecutter — Creating custom reusable project templates | by Matias Calderini | Towards Data Science

Your past and future projects are probably all unique in some way. That being said, if you are (and you should be) moderately organized, the underlying structure of it all is not going to change much across projects: a folder for the raw data, another one for the processed data; a folder for source code, a folder for reporting and so on. Having a shared structure helps not only in the short term when trying to find and use every resource related to your project, but also in the long run in terms of consistency and replicability across individual projects.
Good structuring of your projects is time saving when done right, but can be a time sink and a drag when first starting a new project if you are stuck creating README’s and .gitignore’s instead of importing matplotlib. One could always simply copy an older project and remove/modify the old files and folders, but that doesn’t make it any more fun or faster than starting from scratch. Especially if you have to change the project’s title in every document, the author’s name on every contact section of every file, etc. The real best solution for this problem, is to use a flexible templating tool. In truth, there’s a plethora of templating tools, however, one that distinguishes itself for its simplicity, flexibility and intuitiveness is cookiecutter.
If you haven’t yet heard about it, or you haven’t yet taken the time to play around with it to optimize your templates, in this post I’ll show you how to quickly get started with existing cookiecutter templates found online, how to customize them and how to make your own from scratch with no code or complicated setup whatsoever.
Installing cookiecutter is as simple as installing any other package in python:
pip install cookiecutter
Or if you are using conda:
conda install -c conda-forge cookiecutter
I personally prefer having a general purpose conda environment that includes cookiecutter (among other common/quick-use tools), and then create a new environment for the specific project dependencies. But I leave that part up to you.
As with many resources in open-source paradise, you could begin using cookiecutter templates by using one of the maaaaaany public templates that you like.
One that I particularly like is the cookiecutter-data-science template. I highly recommend you visit the link and look at the whole template structure. Not only it is a great directory tree for your files, but it should also help you organize the conceptual flow of general data-related projects. We will learn in the next section how to customize it if needed. If you are curious about what other templates exist for different types of projects (Django templates, data journalism templates, etc.) you should look at the extensive list on the official documentation.
For now, though, we will use cookiecutter-data-science. Let’s imagine that you want your project to be saved in the folder AllProjects, where you normally save all of your projects. Simply head to that directory, activate your virtual/conda environment at the command/Anaconda prompt (if you were not already there) and type:
cookiecutter <repo url>
If indeed you are using cookiecutter-data-science, this becomes:
cookiecutter https://github.com/drivendata/cookiecutter-data-science
or, if you want to save some characters, you can use the shortcut for GitHub addresses:
cookiecutter gh:drivendata/cookiecutter-data-science
If it is the first time you use it, it will first download the template locally. Once that is done is when the cookiecutter action begins. Depending on the template you chose, you will be prompted to answer different questions, such as what the project name is, the author’s name, project description, license, etc. These are all different customization options you can take (or simply accept the defaults). Fill the required answers or pick from the offered options (defaults appear between square brackets).
I’ll assume that (if you were asked the name of the project, which normally should be the case) you chose the name ElProject for your project’s root directory. After you went through all the questions, you should now see in your AllProjects directory, the new project folder with all of the contents offered by the template:
AllProjects└── ElProject    └── <all the template's subfolders and files>
Simple as that!
Depending on the template, you might see that it includes not only a good directory tree, but also useful files such as .gitignore (to exclude files from version control), a pre-written README, setup and requirements files and much more goodies.
If you wanted to start a new project let’s say super secret project, run the command again, in your AllProjects directory, answer the prompts and you should see the added project folder in the directory:
AllProjects└── ElProject    └── <all the project's subfolders and files>└── super secret project    └── <all the project's subfolders and files>
If you need any additional customizations you can always add/remove/change anything in your project’s directory. But what if you were doing the same customizations over and over again. Maybe you prefer to create LaTex reports of every project you do, or maybe a PowerPoint presentation and you are tired of creating the files, copying the same structure from previous projects and then correcting them with your current project’s info. If this is your case, then keep reading onto the next section where I will cover how to create custom templates.
The first step in creating your own cookiecutter template is to create a folder with the name you want our template to have. You should always choose a name that is easy to remember and descriptive of the type of projects that use the template. For this tutorial though, we will do the complete opposite, and name our new template dasTemplate.
This folder can sit anywhere in your computer, but I advise you keep all of your templates together, such as in the folder AllTemplates (if you then save the whole collection of templates in an easily accessible remote repository, e.g. Github, you get extra points). As it is now, we have the following directory:
AllTemplates└── dasTemplate
Now, head into the new empty folder dasTemplate to start creating your template.
The most basic functionality we want to recreate in our template is the ability to create an empty folder named after our project. For this, into dasTemplate add the next two things exactly as they appear (that includes the curly braces):
A .json file called cookicutter.json (if first time, just create a text file, but save it as .json instead of .txt)A folder named {{cookiecutter.project_name}}
A .json file called cookicutter.json (if first time, just create a text file, but save it as .json instead of .txt)
A folder named {{cookiecutter.project_name}}
So that our template directory now looks like this:
AllTemplates└── dasTemplate     ├── cookiecutter.json     └── {{cookiecutter.project_name}}
Before using the template, you need to open the json file and inside of it, write the following:
{"project_name": "default_name"}
Before explaining the two of them, we can see what using this template would give us.
To do so, head to where you want your project to be created (such as the directory AllProjects) and pass the same command as when we used the cookiecutter-data-science template. However, this time instead of providing the repository’s url, you would simply pass the path to your template (unless you want to use your template from GitHub too, then pass the repo’s url...):
cookiecutter path/to/template
For example (if you are running windows as I am):
cookiecutter C:\Users\USer1\Documents\AllTemplates\dasTemplate
When you run this, you will get the following prompt:
project_name [default_name]:
You might recognize this as the pair we wrote in the .json file. For now, simply input your project’s name e.g. ElProject. After that, you should see the new project folder added to the directory:
AllProjects└── ElProject
So, what did we do here? The answer to that question is best separated in two. First, one can think of the cookiecutter.json to be the document of prompts we want the user to receive when creating a new template. As I hinted, the second part of the .json pair is simply the default value that is used if you don’t provide any answer to the prompt. These parameters can, in turn, be used anywhere in the template with the syntax {{cookiecutter.parameter}}. In that sense, when we created the folder {{cookiecutter.project_name}} we are telling cookiecutter to look for the answer to the prompt for project_name and use it as the name for the folder when creating a new individual project.
This means that when I told you earlier you had to copy the basic template perfectly, this is not quite true. In the .json file, you could actually have written something like:
{"bestCheese": "Cambozola"}
And the template structure:
AllTemplates└── dasTemplate     ├── cookiecutter.json     └── {{cookiecutter.bestCheese}}
As long as what you write within the template in double curly braces ( e.g.{{cookiecutter.parameter}}) matches a parameter in the json file, then cookiecutter will do the rest of the work for you.
We can now start growing our template slowly by selecting which files we want our template to recreate in the project’s folder. Doing this in cookiecutter is as intuitive as can be:
Any file and folder you place in the template will appear AS IS in the indivdual projects started with the template. Not only the files and folders will be there, but also their contents. That is, anything written in them, every image, everything!
So let’s say you want your projects to start with a README.md on the root directory and maybe have an aboutme.txt file for a brief description of you and some contact information, as well as a reports folder that has a template for a LaTex report:
AllTemplates└── dasTemplate     ├── cookiecutter.json     └── {{cookiecutter.project_name}}          ├── README.md          ├── aboutme.txt          └── reports              └── monthlyReport.tex
Run the command in the directory where you want the project to be created and you should see:
AllProjects└── ElProject    ├── README.md    ├── aboutme.txt    └── reports        └── monthlyReport.tex
Super easy, but a bit boring. Let’s personalize it more by using some new variable parameters for the file names and their content. Let’s ask for a short description of the project to be added to the README.md for the different projects, the aboutme.txt to be named after the individual project’s author and the project’s name and description to appear on .tex files. We will first modify the .json file to ask for all of this new info:
{"project_name": "default_name","description": "short description","author": "Dr. Myfull Name","email": "reachme@mail.com"}
Now let’s modify the files and folders using the curly braces. The new template directory should look like this:
AllTemplates└── dasTemplate     ├── cookiecutter.json     └── {{cookiecutter.project_name}}          ├── README.md          ├── {{cookiecutter.author}}.txt          └── reports              └── monthlyReport.tex
The README.md could look like this:
# {{cookiecutter.project_name}}This is a great README file by {{cookiecutter.author}} ({{cookiecutter.email}})The project is about:{{cookiecutter.description}}
For the LaTex, the syntax might change a bit (which why I chose it as an example) since latex uses curly braces in its own syntax. Pay attention to the following example of a monthly report:
\documentclass{article}%%%%% Packages %%%%%\usepackage{graphicx}\usepackage{glossaries}%%%%% Graphics %%%%%{% raw %}\graphicspath{{../figures/}}{% endraw %}%%%%% Glossary %%%%%\makeglossaries\loadglsentries{glossary}%%%%% Project details %%%%%\title{ {{cookiecutter.project_name}} }\author{ {{cookiecutter.author_name}} }\date{\today}
The first thing I want you to notice is the Project details section. For the title and author, the double curly braces are separated by blank space from the curly braces that encircle them. That is not purely esthetic. In certain cases, if you were to write it without the spaces, cookiecutter wouldn’t be able to properly create your folders. So I recommend trying out your template step by step when you encounter this type of situations whether you are working in Latex or elsewhere.
The second thing to notice is the Graphics part. Maybe you’ve already asked yourself “what if I need to use double curly braces within my files?”. If that is your case, and you were to proceed normally, cookiecutter (or more properly said, the language on which it is based) would most likely throw an error at you, thinking that you are trying to use a variable or command that most likely doesn’t exist. You can circumvent this problem, simply by surrounding it by the {% raw %} and tags {% endraw %} (one to open and one to close). All text between the raw-endraw will appear on the individual projects, exactly as it was typed in the template; curly braces and all.
This is not the only extra functionality that we can exploit with cookiecutter. We will see more helpful commands in the next section.
So far we saw how to create the replicable templates with what I called “curly braces” syntax. This syntax actually belongs to the templating language Jinja which runs the cookicutter show under the hood. You might be familiar with the scope and applications of the language if you have worked with Django, or Liquid.
A detailed explanation of all the things one can do with Jinja is well beyond the scope of this post. Instead, I’ll just mention one particularly useful basic trick that can help boost your templates.
Suppose, you normally work in three different teams and you want your template to specify to which team the individual project actually belongs, along with the list of names of each team. You could simply create a new parameter (or context variable in Jinja lingo) in your .json called “project_team” and fill in the name of the team at every use of the template. However, you could misspell the name which would result in unexpected and/or annoying errors. On top of that, if you were to simply write the team name every time, how would you update your files as a function of your answer?
What you can do is set up the .json to offer a selection of options to choose from instead of a fully open field and use good ol’ if statements to tell the template which team’s list to print based on your answer. To do this, first modify the .json file by simpy adding all the options between brackets where you would normally put the default answer (by default now the first option in brackets will be taken):
{"project_name": "default_name","description": "short description","author": "Dr. Myfull Name","email": "reachme@mail.com","project_team": ["Great Team", "Team of Legends", "The Other Team"]}
Now let’s add to the README.md the team members list based on our answer to the prompt:
# {{cookiecutter.project_name}}This is a great README file by {{cookiecutter.author}} ({{cookiecutter.email}})The project is about:{{cookiecutter.description}}## Team:{% if cookiecutter.project_team == 'Great Team' %}* Max E.Mumm* Jack Pott{% elif cookiecutter.project_team == 'Team of Legends' %}* Armando E. Quito* Al E.Gater{% elif cookiecutter.project_team == 'The Other Team' %}* May B.Dunn* Justin Thyme{% endif %}
If you run the command to create a new project, you should see that depending on your choice of team, the list of team members of the README.md will be different. For example, if I worked with the “Great Team”, the README in the project’s folder would look like this:
# ElProjectThis is a great README file by Dr. Myfull Name (reachme@mail.com)The project is about:short description## Team:* Max E.Mumm* Jack Pott
With just this added functionality, you should now be able to create very flexible templates that will allow you to quickly start new projects with the same shared structure without needing to change all the little details.
Hopefully, if you read all the way down, you now have a good idea of how to work with cookiecutter templates. You can now use pre-made ones, customize them to your needs or create your very own ones. If you would like me to introduce more powerful Jinja commands to further boost your productivity or have any comments/questions, feel free to leave a comment or join me on social media. Happy templating, I’ll go have some cookies now...
Originally published at https://maticalderini.github.io/blog/tutorial/2020/04/13/cookiecutter.htmlon April 13, 2020.