How to Structure Your Data Science Workflow | by Mikhail Klassen | Towards Data Science

From the outside, data science can appear to be a huge and nebulous discipline. Today’s data science experts did not attend university to get data science degrees (although many universities now offer these programs).
The first generation of professional data scientists are drawn from the disciplines of mathematics, statistics, computer science, and physics.
The “science” part of data science is the classic work of posing a question, generating hypotheses, examining the evidence, and formulating a model that explains the evidence.
These are skills that anyone can learn, and there are more resources than ever to get started.
One of the best resources is Kaggle. Their data science competitions are a chance for anyone to get their feet wet with a real project. The community that has formed around these challenges is also a great place to learn from others.
When I transitioned from physicist to data scientist, Kaggle was one of the resources I used when teaching myself new skills, especially the use of machine learning libraries like scikit-learn.
In this article, I’ll be using the classic challenge “Titanic: Machine Learning from Disaster” to explain how to approach any data science problem and find a winning solution.
The aim of this challenge is to build a model that can predict the survival of a passenger, based on information known about them from the passenger manifest. This is based on historical data, and we know things like the names, age, gender, ticket class, and family information for many of the passengers — and whether they survived the disaster or not.
Kaggle provides training data and test data. The training data has the “ground truth” label of survival (yes/no), but the test data does not include the ground truth label. Kaggle retains those labels and uses them to score your submission. The test-data predictions are up to you to predict, and the accuracy of your predictions are used to determine your place on the leaderboard.
A side note about the Titanic challenge: if you look at the leaderboard, you’ll see a lot of perfect scores. This naturally leads you to wonder, “how did they do that?”
The answer is disheartening — they cheated. If you google around for a while, you’ll discover that the full test data with ground truth labels is available on the internet. Those with perfect scores simply submit the true labels instead of the predictions of a machine learning model... and receive a perfect score.
But they have failed the true test — these challenges exist for mastering a craft, not stealing a high score.
Before starting your data science project, I recommend setting up your work environment thus:
New project folder, with a subfolder for storing dataSeparate virtual environment, with your standard data science libraries installed.
New project folder, with a subfolder for storing data
Separate virtual environment, with your standard data science libraries installed.
For the virtual environment, I recommend using conda to manage your Python environment. My preferred data science libraries are numpy, pandas, matplotlib, seaborn, and scikit-learn. Depending on the nature of the problem, other libraries (like scipy) may be relevant. Deep learning challenges will involve installing either Tensorflow or PyTorch.
For this data science exercise, we won’t need any deep learning tools, but if you’re curious, I wrote up a guide on setting up a deep learning data science environment in PyTorch.
towardsdatascience.com
Finally, let’s load the data. Assuming you have downloaded the data for the challenge from Kaggle onto your own machine into a subfolder called data and you are writing code inside a Jupyter notebook.
Alternatively, you can create a data science notebook directly on Kaggle’s platform.
It’s important not to skip this first step.
Whenever you work with new data, it’s important to understand what the data contains, what the variables mean, what units and data types are being used, and what the distributions look like.
This will help you develop an intuition for the data and make it easier to generate hypotheses, which hopefully also makes the solution easier to find.
The challenge website explains the data well, with a table explaining each variable:
Most of these are self-explanatory, but sibsp and parch warrant a bit more information:
sibsp: The dataset defines family relations in this way...Sibling = brother, sister, stepbrother, stepsisterSpouse = husband, wife (mistresses and fiancés were ignored)
parch: The dataset defines family relations in this way...Parent = mother, fatherChild = daughter, son, stepdaughter, stepsonSome children travelled only with a nanny, therefore parch=0 for them.
The seaborn Python library excels at visualizing distributions, so in this section, I’m going to inspect the data in a few different ways.
I’m interested to know whether our assumptions about the survival rate of women and children hold up, so I created the following plot.
Our assumptions hold up fairly well, but it’s notable that quite a few children did not survive, and quite a few men of all ages did indeed survive. Could their survival be related to something else?
Perhaps the passenger class is predictive of survival. Let’s take a look at the data again, but this time splitting out by class.
We see a few things here. Of the adult men that survived, a higher fraction of 1st-class passengers survived. There were adult male survivors among 2nd and 3rd class passengers as well, but not as many relative the number of the passengers in each group.
Of the women that did not survive, most of them were 3rd-class passengers.
This tells us that sex, age, and passenger class are all likely to be predictive of survival, but that there are outliers in each group. It’s not clear whether this is random, or due to more subtle factors.
Finally, let’s take a quick look at where these passengers boarded. Perhaps that might tell us something.
The letters S, C, and Q stand for Southampton, Cherbourg, and Queenstown. The majority of passengers embarked at Southampton. Passengers embarking at Cherbourg appear to have a slightly better chance of survival from among their cohort, but there does not appear to be a strong correlation between port and survival.
With these insights, we are already starting to formulate hypotheses about the data, which we’ll test later. Without visualizing the data, we wouldn’t have these same intuitions.
Even under the best of circumstances, data is rarely “clean”, meaning that there could be missing values or mistakes in the data. Other times, data will be recorded in units that need to be converted, filtered, or otherwise processed before any further work can be done.
There is no single way to do data cleaning. It will depend on your data:
Images may need to be rescaled, rotated, color-corrected, smoothed, sharpened.
Audio may need to be filtered, remastered, de-noised, or normalized.
Natural language data (text) may need to be case-corrected, have stop words removed, and punctuation stripped out.
etc.
If we look closely at our Titanic data, we can find a few problems:
This code produces the output:
Column "Age" is missing data.Column "Cabin" is missing data.Column "Embarked" is missing data.
Here I’m using pandas to check for null values with the .isna() method. Missing data or null values produce a “NA” code. If numerical data is expected, it produces a “NaN”, meaning “Not a Number”.
The Titanic data has a lot of missing values in it. Sometimes we don’t know a person’s age, or what cabin they occupied (if any), or what their port of embarkation was.
This leaves us with a few options:
Drop all rows with missing values
Find reasonable values to fill the holes
Each approach has its merits. In the first case, we make no assumptions about the data, and just choose to get rid of any incomplete rows in our table. The advantage is we’re not biasing our future model with our assumption, but at the cost of fewer training examples.
When training a machine learning model, more data is always better. If you have a lot of clean data, it may be fine to throw away any incomplete samples. But if every row in your table is precious, it’s better to find values to plug the holes.
The Titanic data set isn’t very large. We have less than 1000 passengers in our training set. And we may need to further subdivide our training data to validate our models, so that leaves us with even fewer training examples.
Machine learning models need numerical data, but a lot of the Titanic data is categorical. We need to convert this data somehow to numbers.
The Sex column has only two values, female and male. We can remap these to 0 and 1.
train_data.Sex = train_data.Sex.map({‘female’: 0, ‘male’: 1})
Dealing with categorical data that has more than 2 possibilities, e.g. the port of embarkation (which has 3 possible values), is covered below using a technique called “one-hot encoding”.
With a few reasonable assumptions, we can actually fill the holes in our data pretty well.
Age
There are several strategies we could employ here, such as simply imputing the average age of all passengers to the missing values.
But we can do better.
My strategy was to look at the average age of passengers in each ticket class.
This gave the result:
Class: 1 -- Median Age: 37.0Class: 2 -- Median Age: 29.0Class: 3 -- Median Age: 24.0
I was not surprised to discover that first-class passengers tended to be older, and that there was a downward gradient in age as you went into second and then third class.
For all the missing age values, I assigned them the median value based on their passenger class.
You could improve this technique even further by looking at the median age for women and men within each class, and then filling in missing data based on those two variables.
Port of embarkation
There aren’t many missing values here, which is encouraging. The most common port of embarkation was Southhampton, so all else being equal, it’s most likely that a passenger would have boarded there. This was true for passengers of all classes.
From cabin to deck
Many of the rows in our table contain a cabin number. It was initially unclear how to make use of this information, but we can determine the ship deck from the cabin number. For example, “C22” is on Deck C.
Passenger cabins are mostly on Decks B through F. Some information about ship layout can be found here. That same page also indicates where the 1st-, 2nd-, and 3rd-class cabins can be found.
For passengers with a known cabin number, I used it to infer the deck.
For passengers without a cabin number, I used their fare class to infer the most likely deck they occupied.
I created a new column in my data frame called “Deck” and wrote all the inferred deck information there. The “Cabin” column can now be deleted.
The output is:
 Class: 1C          59B          47Unknown    40D          29E          25A          15T           1Name: Deck, dtype: int64 Class: 2Unknown    168F            8D            4E            4Name: Deck, dtype: int64 Class: 3Unknown    479F            5G            4E            3Name: Deck, dtype: int64
My strategy here was to look at the deck layout and see where most of the 1st, 2nd, and 3rd class cabins were. It appeared to be decks C, E, and F, respectively, though I may be wrong.
For all the passengers with an unknown deck, I assigned them to a deck based on their passenger class.
I spent a good amount of time investigating what information could be gleaned from the values in the ticket column.
You’ll notice that some tickets have a prefix, like “S.C./PARIS”, followed by a number. Both the prefix and number could tell us something. My guess is that the prefix indicates the ticket vendor. From the ticket number itself we can sometimes infer groups of people traveling together.
I did a bunch of deep cleaning and disambiguation on the prefix data, but in the end, I dropped it, since it didn’t seem to be leading to anywhere. Please comment if you found a way to use this information to improve your models.
There’s a good discussion about this topic on the Kaggle forum.
Now that we’ve cleaned our data, we could try out a few simple tests. Let’s split off some of our own test data that we can use to test our hypotheses. For this split-off test data, we know the ground truth labels, so we can measure the accuracy of our predictions.
We know that survivors of the Titanic fled on lifeboats, and these (we assume) would be filled preferentially with women and children. How accurately can we predict survival from just these two variables?
We’ll use logistic regression to test it:
Which gives an accuracy of
0.7847533632286996
78% accuracy is pretty good! Clearly these two variables are highly predictive, as expected.
Next, we may assume that first class passenger, because of their status or proximity of their cabins to the upper decks, could have more likely been among the survivors, so let’s see if class alone is a good predictor. Then we’ll see if combining it with age and sex improves the previous result.
I had to do some one-hot encoding of the Pclass variable. I explain what one-hot encoding is below and why it’s important. From these tests, I get the results:
Using Pclass as the sole predictor, our accuracy:0.6995515695067265Using Pclass, age, and sex as predictors, our accuracy:0.7937219730941704
So using the passenger class as the only predictor, our logistic classifier gets almost 70% accuracy. Combining with age and sex, we improve slightly on the previous result: 79% vs 78%. This difference isn’t great and could be noise.
What these first few experiments are telling us is that survival will depend in large part on age, sex, and socio-economic status. These three factors alone could probably get us a reasonably good prediction of survival.
But to eke out those last few percentage points of accuracy, it’s going to take some feature engineering.
Really good feature engineering is what often distinguishes the experts from the novice data scientists. Anyone can take an off-the-shelf software library, train a machine learning model in a few lines of Python, and use it make predictions. But data science is about more than just model selection. You need to give that model high-quality predictive features to work with.
Feature engineering typically means creating new features to help your machine learning model make better predictions. There are tools for automating this process, but it’s better to first think deeply about the data and what other factors may be responsible for the target outcome.
In our Titanic example, we have some information families traveling together. The columns “sibsp” and “parch” tell us about the numbers of siblings, spouses, parents, and children that a passenger has. We could create a new variable called “Family Size” that is the sum of “sibsp” and “parch”.
X['Family Size'] = X['SibSp'] + X['Parch']
Many Kagglers will also create a variable called “not_alone”, which is just a binary identifier describing whether a passenger is traveling by themselves.
This particular data set contains a lot of categorical data. Consider the port of embarkation. There are 3 possible values: Cherbourg, Queenstown, and Southampton. ML models need numerical data, so we could map the port to a number:
{'Cherbourg': 1, 'Queenstown': 2, 'Southampton': 3}
But think about how that looks to a machine learning model. Is Southampton 3x greater in value than Cherbourg? No, that’s absurd. Each port matters equally.
We instead perform a “one-hot encoding” of this categorical data, which will create three new columns, one for each port, and we will use the number 0 or 1 to indicate whether the passenger embarked at a particular port or not.
We can do the same for other categorical variables, such as deck. Sex is a categorical variable in our data set, but since the only values in our data set are “female” and “male”, we just use 0/1 to indicate. No need to create new columns.
One major disadvantage of one-hot encoding is that it can create many new columns. Each column is considered a separate feature. More features aren’t always a good thing. You want the number of examples in your data to greatly exceed the number of features. This can help to prevent overfitting.
Some Kagglers find that creating separate “bins” for age or fare ranges is helpful. Consider that when filling life boats, the crew are probably not asking for age, but considering age categories such as “infant”, “child”, “young”, “old”. You can create similar bins and see if this helps your model. I’m going to leave the age and fare variables as they are.
The above steps are actually the hardest part and represents about 80% to 90% of the work.
The next few steps are usually easier and more fun. We can play around with different machine learning models to see how well they perform, and pick a promising one for further optimization.
Since we are simply trying to predict a binary variable, “survival”, any binary classifier will work. If you’re using scikit-learn, you have many choices.
Some of the most popular classifiers are:
Logistic regression
Decision tree
Random forest
Adaptive boosting (AdaBoost)
XGBoost
That last one, XGBoost, isn’t part of scikit-learn, so you’ll have to install it separately.
Let’s take our training data, select a classifier, and test it using k-fold cross-validation.
Cross-validation is a technique whereby a small portion of the data is left out, while the model is trained on the remaining data. The accuracy of the model is then tested against the left-out data. This process is repeated k times, where the portion of left-out data is drawn randomly each time.
The point of this technique is to help avoid overfitting. In your zeal to engineer the perfect model with the highest accuracy, you may accidentally create a model that doesn’t generalize to data outside of your sample. So you always need to be testing your models against data outside of your sample.
Of the classifiers mentioned above, logistic regression and decision trees are the easiest to understand. Random forests are ensemble models constructed from many decision trees. They tend to do well out of the box. AdaBoost and XGBoost are newer, more advanced models. XGBoost is very popular with Kagglers.
I won’t cover the mechanics of each classifier in this article, but that information is pretty easy to find.
My assess_model function uses 5-fold cross-validation to test the accuracy of each classifier on both the training set and the test set. The true worth of a model is always how well it performs on the test set. The accuracy of each one:
# Logistic regressionAverage Train Accuracy: 0.802 ±0.01Average Test Accuracy: 0.780 ±0.02# Decision treeAverage Train Accuracy: 0.974 ±0.00Average Test Accuracy: 0.781 ±0.02# Random forestAverage Train Accuracy: 0.974 ±0.00Average Test Accuracy: 0.791 ±0.03# AdaBoostAverage Train Accuracy: 0.832 ±0.00Average Test Accuracy: 0.804 ±0.02# XGBoostAverage Train Accuracy: 0.964 ±0.00Average Test Accuracy: 0.820 ±0.03
It’s important to note that we’ve used the “out-of-the-box” version of each of these classifiers. These use default values for all of their internal tunable parameters.
Notice how a classifier sometimes gets ~97% accuracy on the training set. That looks amazing, but it’s overfitting. The test accuracy is lower and it’s the variable we care more about.
Of these classifiers, XGBoost has the highest test accuracy. This is one of the reasons why Kaggler’s love XGBoost — great performance with zero tuning.
Most machine learning models have tunable parameters that often influence model accuracy. The best-performing values for these parameters is different for every problem.
In ML, these parameters are often referred to as “hyperparameters” and tuning them is as much art as science.
There are tools out there to help with tuning. TPOT is one example. But keeping it simple, we’re going to perform a simple grid search and manually test a whole range of plausible hyperparameter values to see which one gives us the best results.
scikit-learn has a convenient tool for performing grid searches.
This code takes some time to run, as it tries all the different combinations of hyperparameters.
You can modify the parameter grid to change the search space. Remember, this technique won’t find the best possible hyperparameter values, only the best combination from within the search space.
With this approach we get:
{'bootstrap': True, 'max_depth': 6, 'max_features': 'auto', 'min_samples_leaf': 2, 'min_samples_split': 4, 'n_estimators': 100}RandomForestClassifier(max_depth=6, min_samples_leaf=2, min_samples_split=4)0.830804623499046
So there you have it. You’d be hard pressed to outperform 83% on the Kaggle Titanic challenge without cheating. I mentioned a few other possible ways of boosting performance, such as age or fare binning, and imputing missing age values based both class and gender. You can try these and see if they boost accuracy.
In conclusion, approaching a data science problem is stepwise process of starting from a clean slate, getting to know your data, cleaning it, and then iteratively testing different models and adding more features until you achieve good model performance. From there, you optimize your best model to squeeze a little more performance out of it.
What happens next? Well if you’re a researcher, you publish your results. If you’re an entrepreneur, you operationalize your model to do something useful that others will pay money for. If you’re a Kaggler, you submit your test predictions for a shot at glory (and sometimes money).
If I missed anything, or you find other techniques for doing even better, let me know in the comments. I hope this has helped you on your data science journey.