Data Science 101. A Step-by-Step Guide | Towards Data Science

IntroductionProblem StatementData CollectionExploratory Data AnalysisFeature EngineeringModel ComparisonResults DiscussionSummaryReferences
Introduction
Problem Statement
Data Collection
Exploratory Data Analysis
Feature Engineering
Model Comparison
Results Discussion
Summary
References
There is a certain trend in all technical processes, and data science is no exception. As you obtain more and more experience in any job, you start to notice a trend, which tends to make the job a little easier. The goal of this article is to make your data science job a little more streamlined because the process that I will outline below applies to every data science use case (or at least most), and for those that it is not 100% applicable for, it can still, hopefully, be useful to you. As you will see, there are six main steps to this process — mainly for the development parts of the model. It is not necessarily for deploying the model into a production environment. This process highlights steps from the problem you are trying to solve to the answer of that problem using data science techniques. Keep reading below, if you would like to know more about the six steps of the data science process.
To build a data science model or utilize a machine learning algorithm, you will need to understand what the problem is. This step can also be called something more along the lines of a ‘business use case’. In this step, you will most likely experience working with stakeholders the most, anyone from data analysts, business analysts, product managers, to your company’s senior executives.
Here is an example of a bad problem statement:
“We wanna predict how many people will buy our product in the year 2022”
Here is an example of a good problem statement:
“The current way of predicting sales is inaccurate”
While the first example makes sense, it does not highlight the problem, it highlighted a possible solution instead. The focus first should always be to understand the problem in its simplest form. From there, we can then present possible solutions using data science techniques and models.
Another part of the problem statement can be the process of defining goals. For example, it will be useful to ask what the current sales prediction accuracy is, the goal accuracy, and hopefully, if the model can or cannot reach that goal accuracy, and what it means to not reach it exactly.
Regarding the holistic data science process described in this article, the data collection process is perhaps the furthest removed step from academia to professional environments. As an example, in an educational course, you may be given a dataset right away that is already processed and explored. For working environments or professional settings, you will have to learn how to acquire that data from an outside source or an internal source within a data table. This step can take quite a bit of time as you will need to explore nearly all data tables in your database, or across databases. The data that you ultimately acquire might use different data from various sources. The final data will eventually be read into a dataframe so that it can then be analyzed, trained, and predicted on.
Here are some possible ways of acquiring data:
from Google Sheets
from CSV files
from Salesforce
JSON files
database tables
from other websites
and much more
This step in the data science process can generally follow the same format. At this point, you will have your main, single dataframe. For the sake of the data science problem, you will need to separate your X features versus your y target variable — what you are trying to predict. These features can span from one to hundreds, or even more, but it is best to start off simple and analyze the main features of your dataset first (the ones you intuitively expect to be significant to the model’s prediction), and then get a glimpse of all of the features next.
You can look at a variety of descriptive statistics that can help to define your data, here are some of the easier and more common ways to describe your data — oftentimes with the pandas library:
df[[‘feature_1’, ‘feature_n’]].head() — first 5 rows of your data
df[[‘feature_1’, ‘feature_n’]].tail() — last 5 rows of your data
df[[‘feature_1’, ‘feature_n’]].describe() — count, mean, std, min, 25%, 50%, 75%, max — giving you a good idea of the distribution of your data and specific features
analyzing missing data — sometimes it is expected
data anomalies
erroneous data — negative values that should not be negative, etc.
If you want a big shortcut, you can use pandas profiling, which shows all of these descriptive statistics about your dataframe, and much more in just a few lines of code [5]:
pandas-profiling.github.io
Now that you have explored your data, you might want to engineer your features. To me, this step of the process should be called “before model feature engineering” — as in you are not using a model to edit your features. With that being said, there are a variety of ways that you expect to work with your features. Some of these include methods like creating new features by simply dividing two features together, to grouping features together for the creation of aggregate features.
Here are some examples of feature engineering:
addition/subtraction/division, etc. of current features to make new features
grouping features to create aggregate features
one hot encoding
target encoding for categorical variables to reduce the dimensionality of your dataframe
As you can see, we have performed several steps before starting to discuss the main ‘data science’ part. In this section, whether you are performing something like regression or classification, it is always best to compare several models before choosing one to update and enhance as your final model(s).
For example, although it might seem obvious to pick a specific machine learning algorithm for your use case, it is best to remove your bias, and obtain a baseline for say, 5 to 10 common algorithms. From there, you can compare the benefits of each — not just the accuracy. For example, you might want to compare the time it takes to train the model, or how expensive it could be, to the requirements of transforming your data.
Here are some common data science model comparison methods:
hand-coding several algorithms and creating a table or dataframe that shows results of the benefits or disadvantages against one another-side-by-side
PyCaret [8]
I tend to compare models in the most baseline approach possible so that I do not get into the weeds too much with a particular algorithm, in case I end up not using it
Before implementing your model into production, you will need to discuss the results with your stakeholder. You will look at what your accuracy means or your loss metric, like RMSE — Root Mean Square Error. These results are oftentimes confusing to people who do not study or employ data science, so it is your job to make them as simple as possible so that stakeholders can make decisions from your results — to move on or not for example (sometimes a complex machine learning algorithm is not the answer to the problem).
Here are some things to keep in mind as you discuss your results:
display results with a visual, in addition to the raw results — use tools like Tableau, Google Data Studio, or create your own in Python
aggregate by certain features to highlight where the error is higher or lower, or accuracy is higher or lower
explain what you would do with more data, more time, or a different algorithm
explain what the results mean on the business and financial side of the company — does this model save money or does it just cost money to create and run?
does your model make the process faster, does it make it better?
does it help the internal users only or does it help the customers of your company's product?
There is much to discuss when incorporating a data science model into a company’s ecosystem. That being said, the final step would be to automate the model by putting it into production.
These main steps are important to most data science projects. The steps that happen after usually involve more machine learning operations — meaning, now that your model is approved, you can implement it into the product by involving software engineers, UX/UI researchers, more product managers, and statisticians for A/B tests. One important question you should ask now that the model makes sense to use, is how will you use it?
does your model cause a negative or positive action from customers down the road— for example?
As you can see, there are a lot of things to think about when incorporating data science at a company, but following these six main steps can set an easy outline and plan of attack for efficiently solving a problem using data science.
To summarize, here are the six steps you can apply to every data science use case:
* Problem Statement* Data Collection* Exploratory Data Analysis* Feature Engineering* Model Comparison* Results Discussion
I hope you found my article both interesting and useful. Please feel free to comment down below if you follow this main process for data science implementation — do you agree or disagree, which steps would you remove or add? Do you think it will be beneficial if you do implement this process moving forward in the future?
Please feel free to check out my profile and other articles, as well as reach out to me on LinkedIn. I am not affiliated with any of the mentioned companies.
[1] Photo by Green Chameleon on Unsplash, (2015)
[2] Photo by Emily Morter on Unsplash, (2017)
[3] Photo by Tobias Fischer on Unsplash, (2017)
[4] Photo by Firmbee.com on Unsplash, (2015)
[5] Pandas Profiling — Simon Brigman, Pandas Profiling Homepage, (2021)
[6] Photo by Jeswin Thomas on Unsplash, (2020)
[7] Photo by Robert Anasch on Unsplash, (2018)
[8] Moez Ali, PyCaret Homepage, (2021)
[9] Photo by Ali Saadat on Unsplash, (2020)