Create a Codebase Your Colleagues Will Understand, in 1 Minute | by Lucy Rothwell | Towards Data Science

In organisations, data science codebases (aka repositories or packages) have a common file structure.
Structure refers to the layout of files in the codebase, for example:
In university, Kaggle or pet projects, you can go a bit rogue on codebase structure — after all, the main thing that matters is the output.
But in organisations, how the codebase is structured also matters, and it matters BIG TIME.
What we discuss here are common best practices for structuring codebases — the kind that organisations will expect you to use as a Data scientist.
It matters because in organisations, other people interact with your code. And when other people interact your code, you want them to understand your codebase it in as little time as possible so they can start contributing to it.
If the file structure is in a common format, your fellow data scientists and developers will spend just a few minutes understanding it before being able to get to the fun part — digging into the code itself.
Alternatively, if the codebase structure isn’t in a common format, every time someone looks at a new codebase they would have to spend extra time (say 2 hours) familiarising themselves with the structure of it. Boring right? We’d rather just get to the code and start programming!
Then there’s time-loss to the team overall. If you consider a business that has 100 developers, each working on an average of 10 existing codebases in a year, and they spend 2 hours familiarising themselves with the structure of each – 2000 hours will be spent just understanding repo structure. That’s 83 days or 3 months work lost unnecessarily each year.
The fabulous thing is, this can be prevented if developers spend 1 minute, setting up a common structure for the repo. For 10 repos, that’s a cost of 10 minutes to save 3 months of lost time.
What a bargain!
The good news is, it’s literally as easy as this:
your_directory$ pip install pyscaffoldyour_directory$ putup your_repo_name
And that’s it! After executing these two lines of code in the directory in which you want to store you repo, it will have been created in a file structure that is common to most data scientists – in less than 1 minute! It will look like this and is immediately ready to start coding in:
There are 3 parts of the standard files in your new codebase which help with this – the src folder, the setup.cfg file (and pre-commit hooks) and the README file:
The most important part of this repo is the src (source) folder as this is where your data science code will go. You can build a full data science project from src without touching any of the other files.
The src folder will look like this initially:
You can ignore the __init__.py and skeleton.py files for now. What matters is what you add to this folder — i.e., functionality for your data cleaning, labelling, model etc.
A common way to structure the src folder is as follows:
Files containing functions
A runner file which calls these functions sequentially and provides your output
Here is an example src structure:
As you can see, there are three folders each of which contain python files with functions:
Preprocessing: for cleaning your data (dealing with missing values, standardising the data etc)Models: any models that need built (such as a Bayesian Neural Network) and functions for training it if neededResults: for outputting the results of your predictions (in the form of plots or CSVs etc)
Preprocessing: for cleaning your data (dealing with missing values, standardising the data etc)
Models: any models that need built (such as a Bayesian Neural Network) and functions for training it if needed
Results: for outputting the results of your predictions (in the form of plots or CSVs etc)
We then have the all-important runner.py file which imports data, calls these functions in order, and provides the output (and probably saves your results to a csv or plots to a folder etc).
The README file is another important part of your repo. This tells your fellow Data Scientists how to use your code base. It’s very useful and explains things like:
What your repo does
What dependencies are needed and any installation instructions
Any other instructions for use
How to get in touch with the author
Your colleagues will likely go to this file first when starting work on your code base so it’s worth making this clear
This final step is where we tell your colleagues which installations they need to run your codebase and configure rules (hooks) which help make sure only well-formatted code can be submitted to your codebase.
Firstly let’s look at the setup.cfg file. The main section here worth editing is “install_requires”. This is a super useful section as this tells new users which packages and versions are needed to run your codebase. Without this, your colleagues may install old versions that don’t have the required functions (and as such will cause errors when running).
Libraries are added to this section in the form library == version for example:
In the install_requires section, we can also add three top-quality packages which help make your actual code more readable for your colleagues:
Pre-commit and pre-commit hook are great packages! They essentially host a set of formatting “hooks” and formatting rules to which your code must adhere before you can commit anything. These rules are standard best practices which are widely used (and expected) in data science. For example:
Code line length cannot be greater than 100 characters
Import statements must be at the top of files
Two blank lines must be left between each function
When you try to commit, the pre-commit package will let you know which of the rules have not been met so you can fix them.
Two easy steps for setting up pre-commit hooks:
Install pre-commit by executing the following in your terminal (or other shell) in your repo:
Install pre-commit by executing the following in your terminal (or other shell) in your repo:
your_directory$ pip install pre-commityour_directory$ pre-commit install -t pre-commit
2. Create a .pre-commit-config.yaml file to store your hooks
This file is what pre-commit reads when you commit something — so you should list here which hooks you want. Add the .pre-commit-config.yaml file to your root folder (i.e., the folder above src).
Below is a basic config you can copy into your .pre-commit-config.yaml file to start with — it instructs the three hooks in the bullet points above: trailing whitespace, reorder python imports, flake8 (code style guide enforcement — very widely used).
Code for copy and paste:
default_language_version:     python: python3repos:    - repo: local      hooks:         - id: reorder-python-imports           name: reorder-python-imports           entry: reorder-python-imports           language: python           types: ["python"]         - id: flake8           name: flake8           entry: flake8           language: python           types: ["python"]        - id: trailing-whitespace           name:  trailing-whitespace           entry: trailing-whitespace-fixer           language: python           types: ["python"]
It should look like this:
Now when you commit, the hooks should kick in :) to test, try to commit a .py file with erroneous code in it — it should be rejected and a list of code lines needing fixed should be returned in the terminal.
You can add as many hooks as you like in the .pre-commit-config.yaml file. The documentation for pre-commit can be seen here and a more complete list of hooks can be seen here.
And that’s it! You can push this codebase to GitHub as normal, safe in the knowledge you’ve covered the basics of professional codebase set up :)
They key to structured repos being useful is that everyone uses the same structure — a known one that is familiar. Your organisation will probably have a set structure and it may differ from the one above. However, the above method is a well-known structure in data science and is a good bet the absence of other guidelines.
I hope that’s useful! Feel free to direct message me on LinkedIn if you have any questions. You can also subscribe to my posts here.
Lastly, if you’d like to further support the writers on Medium you can sign up for membership here.