How I scored in the top 8% of Kaggle’s Titanic Machine Learning Challenge🚢
By: Syed Nauman
Tags: Kaggle, Titanic, Board Infinity, Kaggle Competition, Medium

A step-by-step tutorial from exploratory data analysis to model training and fine-tuning

The sinking of the Titanic is one of the most infamous shipwrecks in history.

On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew. While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others. In this challenge, the goal is to build a predictive model that answers the question: “what sorts of people were more likely to survive or die?” using passenger data (i.e. name, age, gender, socio-economic class, etc.)

I have been playing with the Titanic dataset for a while, and I have achieved an accuracy score, back in the time, of 0.78708 on the public leaderboard.

This post is the opportunity to share my solution with you.

To make this tutorial useful so that anyone could benefit, here are the steps I have followed:

Data exploration and visualization: an initial step to formulating hypotheses

Data cleaning and feature engineering

Feature selection

Hyperparameters tuning

Model building

prediction

Submission

— Environment setup

Throughout this Jupyter notebook, I will be using Python at each level of the pipeline. The main libraries involved in this tutorial are:

Python Programming

Jupyter Notebook

Pandas for data manipulation and ingestion

for data manipulation and ingestion Matplotlib and seaborn for data visualization

and for data visualization Numpy for multidimensional array computing

for multidimensional array computing sklearn for machine learning and predictive modeling

for machine learning and predictive modeling Exploratory Data Analysis

Machine Learning

A very easy way to install these packages is to download and install the conda distribution that encapsulates them all. This distribution is available on all platforms (Windows, Linux, and Mac OSX).

— Exploratory data analysis

As in different data projects, we’ll first start diving into the data and build up our first intuitions.

Data extraction: we’ll load the dataset and have a first look at it.

Cleaning: we’ll fill in missing values.

Plotting: we’ll create some interesting charts that’ll spot correlations and hidden insights out of the data.

Two datasets are available: a training set and a test set. We’ll be using the training set to build our predictive model and the testing set to score it and generate an output file to submit on the Kaggle evaluation system.

We’ll see how this procedure is done at the end of this post.

Now let’s start by loading the training set.

We have:

891 rows, 12 columns for train data set

418 rows, 11 columns for test data set

Pandas allow you to have a high-level simple statistical description of the numerical features. This can be done using the describe method.

The count variable shows that 177 values are missing in the Age column.

One solution is to fill in the null values I have used missingno for plotting the missing value, what is missingno? A missingno(nullity matrix) provides a small toolset of flexible and easy-to-use missing data visualizations and allows you to get a quick visual summary of the completeness of your dataset. we’ll see how, pip install missingno(install before importing)

The msn.matrix nullity matrix is a data-dense display which helps you to quickly visually pick patterns in data completion

msno.matrix depicts value in data-dense display, more than 80% of the “Cabin” data is missing and few “Age”rows are also missing

PassengerId: and id that is given to each traveler on the boat

Pclass: the passenger class. It has three possible values: 1,2,3 (first, second and third class)

The Name of the passenger

The Sex

The Age

SibSp: number of siblings and spouses traveling with the passenger

Parch: number of parents and children traveling with the passenger

The ticket number

The ticket fare

The cabin number

The embarkation. This describes three possible areas of the Titanic from which the people embark. Three possible values S, C, Q

EXPLORATORY DATA ANALYSIS

Let’s now make some charts.

Let’s visualize survival based on Survived

The Survived variable based on sex seems to be a discriminative feature. Women are more likely to survive.

Now, we see that:

The age conditions the survival for male passengers:

Younger male tend to survive

A large number of passengers between 20 and 40 succumb

The age doesn’t seem to have a direct impact on the female survival

Let’s now see how the sex affects survival.

As we saw in the chart above and validate by the following:

Women survive more than men, as depicted by the larger female “Women and children first !”.

from the result, it can be approximated that the survival rate of men is around 19% and that of women is around 75%. Therefore, whether a passenger is a male or a female plays an important role in determining if one is going to survive.

Let’s now see how the fare affects survival by binning

qcut : “Quantile-based discretization function. "This basically means that qcut tries to divide up the underlying data into equal sized bins¶

Extracting the passenger titles

When looking at the passenger names one could wonder how to process them to extract useful information.

If you look closely at these examples:

Braund, Mr. Owen Harris

Owen Harris Heikkinen, Miss. Laina

Laina Oliva y Ocana, Dona. Fermina

Fermina Peter, Master. Michael J

You will notice that each name has a title in it! This can be a simple Miss. or Mrs. but it can be sometimes something more sophisticated like Master, Sir or Dona.

Let’s see how we’ll do that in the function below.

Let’s first see what the different titles are in the train set.

Let’s now combine the SibSp and the survival.

Ticket

Let’s now combine the fare and the survival.

Passengers with cheaper ticket fares are more likely to die. passengers with more expensive tickets, and therefore a more important social status, seem to be rescued first.

Let’s now combine the cabin_num and the survival.

Let’s now see how the embarkation site affects survival.

the correlation of survival with cabin_num is -0.06

It seems that embarkation S has a wider range of fare tickets and the passengers who pay the highest prices are those who survive.

We can also see this happening in embarkation c and less in embarkation Q.

Let’s now stop with data exploration and switch to Feature Engineering

FEATURE ENGINEERING

Processing the Names

As we notice that each name has a title in it! This function drops the Name column since we won’t be using it anymore because we created a Title column.

Then we encode the title values using a dummy encoding.

Age

We have seen that the Age variable was missing 177 values. This is a large missing number . Simply replacing them with the mean or the median age might not be the best solution since the age may differ by groups and categories of passengers.

This data frame will help us impute missing age values based on different criteria.

Processing fam_size and Ticket_len

fam_size, This part includes creating new variables based on the size of the family (the size is, by the way, another variable we create).

This creation of new variables is done under a realistic assumption: Large families are grouped together, hence they are more likely to get rescued than people traveling alone.

Let’s first see how the different ticket prefixes we have in our dataset

Processing cabin

Processing the Fare

Let’s impute the missing fare value by the average fare computed on the train set

This function simply replaces one missing Fare value by the mean.

Modeling

In this part, we use our knowledge of the passengers based on the features we created and then build a statistical model. You can think of this model as a box that crunches the information of any new passenger and decides whether or not he survives.

There is a wide variety of models to use, from logistic regression to decision trees and more sophisticated ones such as random forests and gradient boosted trees.

We’ll be using Random Forests. Random Forests has proven a great efficiency in Kaggle competitions.

Break the combined dataset in the train set and test set. Use the train set to build a predictive model. Evaluate the model using the train set. Test the model using the test set and generate an output file for the submission.

Keep in mind that we’ll have to reiterate on 2. and 3. until an acceptable evaluation score is achieved.

Hyperparameters tuning

As mentioned at the beginning of the Modeling part, we will be using a Random Forest model. It may not be the best model for this task but we’ll show how to tune. This work can be applied to different models.

Random Forest is quite handy. They do however come with some parameters to tweak in order to get an optimal model for the prediction task.

Now that the model is built by scanning several combinations of the hyperparameters, we can generate an output file to submit on Kaggle.

— Conclusion

In this article, we explored an interesting dataset brought to us by Kaggle.

We went through the basic bricks of a data science pipeline:

Data exploration and visualization: an initial step to formulating hypotheses

Data cleaning and Feature engineering

Feature selection

Hyperparameters tuning

Model building

prediction

Submission

Future Scope