Structure Your Data Science Projects | by Baran Köseoğlu | Towards Data Science

Data science has some key differences, as compared to software development. For example, data science projects focus on exploration and discovery, whereas software development typically focuses on implementing a solution to a well-defined problem. The ambiguities rarely occur in defining the requirements of a software product, understanding the customer needs, while even the scope may be changed for a data-driven solution.
You can find many other differences between data science and software development, however engineers in both fields need to work on a consistent and well-structured project layout. Afterall data science projects include source code like any other software system to build a software product which is the model itself. Structuring the source code and the data associated with the project has many advantages. The main benefits of structuring your data science work include:
Collaboration across the data science team becomes easier. When everyone in the team follows the same project structure, it becomes very easy to detect the changes made by others and contribute to these changes.Reproducibility. Models should be reproducible not only to keep track of the model versioning but also to easily revert back to older versions in case of model failures. Documenting your work in a reproducible fashion makes it possible to determine if the new model performs better than the previous ones.Efficiency. I have many times checked the old jupyter notebooks to reuse some of the functions for the new projects. I can tell by experience that iterating through an average of ten notebooks, assuming you have a good memory, to find a 20 line piece of code can be frustrating. Submitting the code I have written in a consistent structure avoids self-repeating and duplication.Data management. Raw data should be separated from interim and processed data. This will ensure that any team member working on the project can easily reproduce the models built. The time spent to find the respective datasets used in one of the model building stages is highly reduced.
Collaboration across the data science team becomes easier. When everyone in the team follows the same project structure, it becomes very easy to detect the changes made by others and contribute to these changes.
Reproducibility. Models should be reproducible not only to keep track of the model versioning but also to easily revert back to older versions in case of model failures. Documenting your work in a reproducible fashion makes it possible to determine if the new model performs better than the previous ones.
Efficiency. I have many times checked the old jupyter notebooks to reuse some of the functions for the new projects. I can tell by experience that iterating through an average of ten notebooks, assuming you have a good memory, to find a 20 line piece of code can be frustrating. Submitting the code I have written in a consistent structure avoids self-repeating and duplication.
Data management. Raw data should be separated from interim and processed data. This will ensure that any team member working on the project can easily reproduce the models built. The time spent to find the respective datasets used in one of the model building stages is highly reduced.
Although to succeed in having reproducibility for your data science projects has many other dependencies, for example, if you don’t override your raw data used for model building, in the following section, I will share some of the tools that can help you develop a consistent project structure, which facilitates reproducibility for data science projects.
Cookiecutter
Cookiecutter is a command-line utility that creates projects from project templates. You can create your own project template, or use an existing one. What makes this tool so powerful is the way you can easily import a template and use only the parts that work for you the best.
In this post, I am going to talk more about cookiecutter data science template. Installation is easy and straightforward. To install, run the following:
pip install cookiecutter
To work on a template, you just fetch it using command-line:
cookiecutter https://github.com/drivendata/cookiecutter-data-science
The tool asks for a number of configuration options and then you are good to go.
The project structure looks like the following:
├── LICENSE├── Makefile           <- Makefile with commands like `make data` or    `make train`├── README.md          <- The top-level README for developers using this project.├── data│   ├── external       <- Data from third party sources.│   ├── interim        <- Intermediate data that has been transformed.│   ├── processed      <- The final, canonical data sets for modeling.│   └── raw            <- The original, immutable data dump.│├── docs               <- A default Sphinx project; see sphinx-doc.org for details│├── models             <- Trained and serialized models, model predictions, or model summaries│├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),the creator's initials, and a short `-` delimited description, e.g.│                         `1.0-jqp-initial-data-exploration`.│├── references         <- Data dictionaries, manuals, and all other explanatory materials.│├── reports            <- Generated analysis as HTML, PDF, LaTeX, etc.│   └── figures        <- Generated graphics and figures to be used in reporting│├── requirements.txt   <- The requirements file for reproducing the analysis environment│├── setup.py           <- makes project pip installable (pip install -e .) so src can be imported├── src                <- Source code for use in this project.│   ├── __init__.py    <- Makes src a Python module│   ││   ├── data           <- Scripts to download or generate data│   │   └── make_dataset.py│   ││   ├── features       <- Scripts to turn raw data into features for modeling│   │   └── build_features.py│   ││   ├── models         <- Scripts to train models and then use trained models to make│   │   │                 predictions│   │   ├── predict_model.py│   │   └── train_model.py│   ││   └── visualization  <- Scripts to create exploratory and results oriented visualizations│       └── visualize.py│└── tox.ini            <- tox file with settings for running tox; see tox.testrun.org
Folders
The generated project template structure lets you to organize your source code, data, files and reports for your data science flow. This structure easies the process of tracking changes made to the project. There are five folders that I will explain in more detail:
Data. Data should be segmented in order to reproduce the same result in the future. The data that you have today to build your machine learning model may not be the same data that you will have in the future, ie. the data may be overwritten, or lost in the worst case. In order to have reproducible machine learning pipelines, it is very important to keep your raw data immutable. Any progress made on the raw data should be properly documented and that’s where the data folder comes into play. You don’t need to name your files as final_17_02_2020.csv, final2_17_02_2020.csv anymore to keep track of the changes.
Models. Models are the end products of a machine learning pipeline. They should be stored under a consistent folder structure to ensure that the exact copies of the models can be reproduced in the future.
Notebooks. A lot of data science projects are done in Jupyter notebooks which allow the readers to understand the project pipeline. These notebooks are likely to be filled with lots of functions, and code blocks, which makes even the creators forget about the functionality of the code blocks. Storing your functions, code blocks and results in separate folders allows you to segment the project further and easies to follow the project logic in a notebook.
Reports. A data science project not only produces a model but also figures, and charts as part of data analysis flow. These can be parallel lines, bar charts, scatter plots, etc. You should store generated graphics and figures to be able to report them easily when needed.
Src. Src folder is where you put the functions used in your pipeline. These functions can be stashed according to their similarity in functionality like a software product. You can easily test and debug your functions, while using them becomes as easy as importing them in your notebooks.
I modified one of the earlier projects I worked on for illustration purposes of how to utilize this tool. The repository is not optimized for a machine learning flow, though you can easily grasp the idea of organizing your data science projects following the link.
Makefile
GNU make is a tool that controls the generation of executables and non-source files of a program. It utilizes makefiles which lists all non-source files to be built in order to produce an expected outcome of a program. Makefiles help data scientists to set up their workflow immensely. Most of the time after a data science project is delivered, developers have a hard time remembering the steps taken to build the end product. Makefiles help data scientists to document the pipeline to reproduce the models built. This tool, therefore, should be in the toolbox of a data scientist.
Makefile not only provides reproducibility but also it easies the collaboration in a data science team. A team member, who would be setting up the environment and install the requirements using multiple numbers of commands can now do it in one line:
#oldvirtualenv exsource ex/bin/activatepip install -r requirements.txt#newmake requirements
Watermark
Watermark is an IPython extension that prints date and time stamps, version numbers and hardware information in any IPython shell or Jupyter Notebook session. It provides a simple way to keep track of tools, libraries, authors involved in a project. For large projects, using tools like watermark would be a very simple and inefficient method to keep track of changes made.
To install and use watermark, run the following command:
pip install watermark%load_ext watermark
Here is a demonstration of how it can be used to print out library versions. You can find more information in their documentation:
%watermark -d -m -v -p numpy,matplotlib,sklearn,pandas
Final Thoughts
I can tell by experience that data science projects generally do not have a standardized structure. However, the tools I described in this post can help you create reproducible data science projects, which will increase collaboration, efficiency, and project management in your data team.
If you have any questions regarding the post or any questions about data science in general, you can find me on Linkedin.
Thanks for reading!