Building an End-To-End Data Science Project | by George Liu | Towards Data Science

It is often said that the majority of a Data Scientist’s work is not the actual analysis and modeling, but rather the data wrangling and cleaning part. As a result, full-cycle data science projects that involve these stages will be more valuable since they prove the author’s abilities to work independently with real data, as opposed to a given cleaned dataset.
Fully understanding the value of an end-to-end data science project, I always wanted to build one but not able to, until now :)
I have recently finished my Ideal Profiles project. Since it’s a major project that involves many moving parts, I want to document the process and the lessons learned, which is a further learning opportunity (inspired by William Koehrsen’s great post on the value of data science writing).
In my opinion, a full-cycle data science project should include the following stages:
The biggest counter-argument for working on a Kaggle project is often that it’s only focused on the second stage. Therefore, in this project, I made sure that all three stages are covered.
For the first stage, I did web scraping to get the data and since the data was dirty, I had to wrangle to make the data ready for analysis. Then I made various data visualizations and performed analyses as stage two. Lastly, I wrote posts to communicate findings and launch this project into production.
Of course, I could have made this project even more complete by including a machine learning component, e.g. using NLP to classify job postings based on contents, but that will delay the project completion time significantly which brings us to the next point:
Potentially, there can be an unlimited number of things to work on for a given project, but in reality, we only have limited time. To coordinate these two competing factors, we need to be disciplined.
For me, “iterative thinking” really helped — look, Rome wasn’t built in a day, so let’s build something that works and ship it first, and then we can always come back to improve with more features. On the other hand, this also means we need to be able to deal with “non-perfectness” and not get fixated on details.
With this philosophy in mind, I was able to delay some very tempting features and put them in the To-do’s section of the project doc. One of them was to use a larger dataset by scraping data from Indeed USA instead of the Canadian site.
Given the end-to-end nature of the project, there were many different aspects to work on — web scraping, data preprocessing, plotting...If we put all codes in one Jupyter Notebook, it’ll be too long and too complex to manage. I then decided to use Python scripts combined with a central Jupyter Notebook to solve this problem.
I divided supporting functions into three big categories and housed them in three corresponding scripts:
scrape_data.py — containing functions needed for web scraping such as “get_soup()” and “get_urls()”
process_text.py — containing text processing and cleaning functions such as “tokenize_text()” and “check_freq()”
helper.py — containing file I/O and plotting functions, e.g. “plot_skill()”
This way, I was able to maintain a super light and organized central Notebook. Functions are then imported and called from the Notebook as needed like this:
from scrape_data import *from process_text import *from helper import *
As many of the scraping scripts I found online didn’t work, I was determined to make sure that my project is reproducible. Aside from solid code, a robust README document and a complete environment dependency file were also part of the solution.
README.md — I made painstaking efforts to make sure all relevant details are captured, particularly, how to set up the environment and how to use the scripts.
env_ideal_profiles.yaml — by freezing all dependencies into this file, I made sure the user can exactly re-create the same Anaconda Python environment I used. More info is available here.
Good coding practice does matter! In particular, I found the following practices incredibly useful in coding larger and more complex projects:
Write clear, concise and informative comments
Have meaningful and descriptive variable/function names
Provide detailed and structured docstrings
Ensure exception handling using Python “try except” block
These things may seem trivial when you have a project that is a 30-line Jupyter Notebook, but can really be critical when dealing with a major project that needs hundreds of lines of code!
I used to feel comfortable with just basic Matplotlib skills. However, for this project, I not only needed to combine several plots into one but also had to do detailed customization such as rotating axis tick labels...at that point, basic Matplotlib skills simply won’t suffice anymore.
It actually turned out to be a great opportunity to learn Matplotlib. Once I learned what it can do, I found it impossible to look back, simply because, Matplotlib is really powerful! Its object-oriented approach allows you to modify almost anything...Check out the below tutorials to find out:
Matplotlib Tutorial: Python Plotting
Effectively Using Matplotlib
Python Plotting With Matplotlib (Guide)
That’s all, the final blog post about the Ideal Profiles project I recently completed. You can find the other two articles here and here.
What’s your thought? Anything I missed or I could improve on? Please feel free to comment below. I welcome any feedback.
Thank you for reading!