Should I Bother With Structure in Python?
By: Dennis Bakhuis
Tags: Machine Learning, Pyhton, Programming, Data Science, Planning

Should I Bother With Structure in Python?

Formatting and structuring your Python project when needed

Structure can surely help! (Image by Johnny Gutierrez from Pixabay)

In the last three months I wrote a Python course in 10 minutes bits a day called: learn Python 10 minutes a day. It was fun to write and I got many positive responses through my LinkedIn (which I really appreciate). I also get questions regularly, which I always try to answer.

Last week I got an interesting question from Domenico:

“𝙸𝚜 𝚝𝚑𝚎𝚛𝚎 𝚊 𝚒𝚗𝚍𝚞𝚜𝚝𝚛𝚢 𝚜𝚝𝚊𝚗𝚍𝚊𝚛𝚍 𝚏𝚘𝚛𝚖𝚊𝚝 𝚝𝚑𝚊𝚝 𝚢𝚘𝚞 𝚗𝚎𝚎𝚍 𝚝𝚘 𝚜𝚝𝚛𝚞𝚌𝚝𝚞𝚛𝚎 𝚢𝚘𝚞𝚛 𝙿𝚢𝚝𝚑𝚘𝚗 𝚙𝚛𝚘𝚐𝚛𝚊𝚖𝚜?“

I think this is a great question which can open a great discussion. As usual, the answer starts with ‘it depends’. Here is my personal view:

Python is a multi-purpose programming language and it has incredible many use-cases. Sometimes, it’s just used as glue to connect two different services. Service A has some weird binary format as a result and service B only accepts JSON. Python can solve this problem and ‘glue’ these services together. Even if no changes are needed, services are easily linked into one master service. The use-cases for Python do not have to be simple and it can be a full-fledged service itself. There are many examples such as Dropbox, Quora, Spotify, and Netflix. There are even complete GUI applications created using Python! As there are so many different use-cases, there is definitely not one industry standard format, project layout, or structure. However, each field (and sometimes sub-fields) sure have their layouts, structures, and ways of work.

“The joy of coding Python should be in seeing short, concise, readable classes that express a lot of action in a small amount of clear code — not in reams of trivial code that bores the reader to death.” — Guido van Rossum

Python is flexible and by itself, does not really care about structure. Structure is mostly useful when you collaborate with other people (teams), when a project gets large, or when you want to re-use certain parts of a project.

If you are unsure about structure, you probably do not need it (too much) at the moment.

In my early career I mostly practiced functional programming, all in a single file. For small projects this is perfectly fine. When projects get larger, you will (probably) feel a need to split to code, for several reasons. These reasons might be maintainability or simply getting tired of scrolling through the code. One way of restructuring is to split certain functions into other modules that can be imported later. However, don’t worry too much about it in the beginning. Your understanding of these topics will grow with experience and when you see the need!

It can become a bit different when you are working with particular frameworks. For example, if you want to work with web-frameworks such as Flask or Django, they come with their (somewhat) fixed project layout and way of work. These structures can be confusing in the beginning but have big benefits in the long run. It feels like you almost have to do bookkeeping on where you wrote what piece of the code. After a short while, you get used to the structure and it is actually very helpful. Due to the standardization, all your website or web-apps have the same layout, making it super easy to find bits of code you want to reuse. Also collaborating is easier as all your collaborators work in the same layout. Therefore, while it feels like a burden, it can be picked up quickly and has large benefits in the long run!

A possible structure of a Flask project

When collaborating with other people, it also makes sense to have some structure. For example, in a data science project, it makes sense to structure all the different steps (data collection / EDA / etc.). There are a couple of “standard”, so called cookie-cutters, that help you set up a project layout. These are merely suggestions on how you can structure your project. There are many different project-types available as cookie-cutter. Definitely search for them if you like to have some structure. If you are alone and the project is simple, it can be a bit too much. Still, getting used to some structure in the beginning can help in the long run.

We mentioned a cookie-cutter, which is not the actual layout, but a tool to create projects (and file/folder layouts) in a standardized way. Actually, it is a Python package that creates a directories and files for a new project after asking a couple of questions. The tool itself uses predefined structures that are published on Github specifically created for cookie-cutter. To install cookie-cutter install it in a new environment (not sure how to do that? Here is a great guide!):

pip install cookiecutter

After installing the cookie-cutter tool you can create a new project from the command line. For this you need a cookie-cutter format. A great one for data science is from drivendata and is called Cookiecutter-data-science. The tool will ask you a couple of questions and afterwards create a new project folder and all the structure inside the project. To initiate the process, type:

cookiecutter https://github.com/drivendata/cookiecutter-data-science

The first question is the project name, which will also be used for the project folder. Next it asks for a repository, you can use the default value. Next are the author and a brief description. For a license, if you are not sure, choose a MIT license, which lets every use the code at free will. I think this is better than no license. The last two questions are about Amazon services, which you can leave empty (default). When you are done, it has created a new folder with all the files in place. Pretty easy!

Well, this was a much longer answer than I anticipated. To summarize a all the points:

If you are just starting Python, do not worry too much about it. You will feel the need for structure and layout when your projects get bigger.

When using certain frameworks, they come with their own formatting (i.e. project layout). There is generally not much choice and you have to adapt. However, it is almost certainly worth the investment.

In collaborations it also makes sense to apply structure. You can come up with your own layout or use a predefined cookie-cutter to set it up.

These are just my two cents and I am curious what others think. Feel free to comment below this post or connect on LinkedIn.