Comprehensive Beginner’s Guide to Kaggle & The Titanic Survival Prediction Competition
By: Jason Chong
Tags: Kaggle, Data Science, Machine Learning, Data Visualization, Exploratory Data Analysis

Comprehensive Beginner’s Guide to Kaggle & The Titanic Survival Prediction Competition

If you know me, I am a big fan of Kaggle. God only knows how many times I have brought up Kaggle in my previous articles here on Medium.

But my journey on Kaggle wasn’t always filled with roses and sunshine, especially in the beginning. Due to my lack of experience, I was initially struggling to grasp the workflow behind a machine learning project as well as the different terminologies that were being used.

It was only after months of sifting through online resources which included many technical articles, documentation and tutorial videos that I slowly started to learn concepts like label encoding, cross-validation and hyperparameter tuning.

I wished back then that there was a one-stop-shop where I could learn not only the steps behind a machine learning project but more importantly the rationale behind doing them. So I thought, why not use the knowledge that I have accumulated over the last few months to create that one-stop-shop to help others who might be going through the same thing as I did?

In this article, I will explain what a machine learning problem is as well as the steps behind an end-to-end machine learning project, from importing and reading a dataset to building a predictive model with reference to one of the most popular beginner’s competitions on Kaggle, that is the Titanic survival prediction competition.

Photo by Annie Spratt on Unsplash

This was the project that introduced me to the world of machine learning and also my first competition on Kaggle.

What is machine learning?

Actually, before we even get into machine learning, I think it is important that we first understand the purpose behind building a model.

What is a model and why is it important?

A model is a simulation and simplification of the real-world.

Whether you are an engineer building a skyscraper, a hedge fund analyst picking a stock or even a politician who is running for an election, we all deal with uncertainty in this world. A model allows us to minimise this uncertainty and make more informed decisions.

Now, a model is not perfect and it won’t help us consistently predict the actual outcome of a particular event but we can definitely come very close to it. With large amounts of data, together with the power of artificial intelligence, humans have become increasingly proficient at making high-quality, accurate predictions using models.

This is where machine learning comes in.

Machine learning is when computers learn to derive trends and patterns that are present in data. A machine learning model is “trained” using historical data. Once the model has been built and tested, we can then use it to make future predictions on other data points.

Machine learning models differ from statistical models such that they require a minimal human effort but more importantly, there are fewer assumptions that that go into a machine learning model than a statistical model. Statistical models are mathematics intensive and based on coefficient estimation. It requires the modeller to understand the relationship between different variables before they can be included in the model.

For those reasons stated above, machine learning models not only make more accurate predictions but also predictions that are more robust and far exceed the capabilities of any ordinary model.

Supervised vs unsupervised learning

For the sake of keeping this article as comprehensive as possible for beginners, I want to also highlight the two main branches of machine learning, that is supervised and unsupervised learning. Understanding the distinction between the two branches will help us think more carefully about the type of problem that we are trying to solve using machine learning.

Supervised learning is training a model using labelled data. In other words, we are explicitly telling our model the sample outcome of our predictions.

Supervised learning can be further broken into classification and regression. Classification problems are when the outcome of our predictions and discrete and categorical. Regression, on the other hand, has predictions that are of a continuous distribution.

To illustrate a classification problem, imagine that you are building a spam classifier which helps classify your emails as either spam or non-spam. What you would do is prepare a training set which contains the different features of your emails like their word count, use of particular words and punctuations and most importantly their labels, spam or non-spam. Then, you can train a machine learning model using this training set and the model will start to learn the relationship between the features and their labels. Once the model has been built to your desired accuracy, you can use this classifier to classify your other emails.

Regression differs from classification in that, rather than having a discrete target variable such as spam or non-spam, the outcome in a regression problem is continuous. A perfect example of regression is predicting house prices. Here, house prices is a continuous variable. Again, we need to prepare a training set which contains house observations along with their features such as the number of bedrooms, distance away from the city, living room area and of course, their sale prices. A model will then learn how the different house features correlate with the final sale price. Subsequently, we can use this model to make predictions on other houses.

Now, let’s move on to unsupervised learning.

As we saw in the spam classifier and house price prediction examples, we included the sample outcome (spam vs non-spam and final sale price) as part of our training set to train our machine learning model.

In unsupervised learning, we don’t include any sample outcome but instead simply let the model derive any underlying patterns that are present in the data and group them accordingly.

Since most Kaggle competitions use supervised learning, I won’t go into unsupervised learning in too much detail in this article.

Aim of this competition

The aim of this competition is to build a machine learning model that will help us predict the survival outcome of the passengers on the Titanic.

This is an example of a binary classification problem in supervised learning as we are classifying the outcome of the passengers as either one of two classes, survived or did not survive the Titanic.

More specifically, we would like to investigate how different passenger features like their age, gender, ticket class etc impact their survival outcome.

The evaluation metric of this competition is the percentage of passengers in the test set that are correctly predicted by our classifier.

Notebook walkthrough

In this section, I will briefly discuss the steps and results of my analysis of the competition. You can find the complete code for this project on my GitHub.

Datasets

There are three types of datasets in a Kaggle competition. In more advanced competitions, you typically find a higher number of datasets that are also more complex but generally speaking, they fall into one of the three categories of datasets.

Training set: This is the dataset that we will be performing most of our data manipulation and analysis. In relation to the Titanic survival prediction competition, we want to analyse and/or create features that can help us predict the survival outcome of the passengers. Furthermore, as the name suggests, this is the dataset that we will use to train our machine learning model. Test set: Once our classifier has been built, we want to make sure that it does not merely fit our training data but more importantly able to make predictions on out-of-samples data. Recall the main objective of modelling is to predict real-world data that were not used to train our model. If a model only performs well on training data and unable to make predictions on new data, the model is essentially useless. The test set is what Kaggle will use to assess the accuracy of our model i.e. how accurate our classifier is at classifying passenger data that our classifier has not seen before. The test set has one column less than the training set which is the response (target) variable and in our case, is the Survived column. Sample submission: This is the format in which we want to submit our final solution to Kaggle. Our final dataframe needs to have the same shape (same number of row and columns) as well as the same column headings as the sample submission dataframe.

Data description

Here, I will outline the definitions of the columns in dataset. You can find this information under the data tab of the competition page.

Survived: 0 = Did not survive, 1 = Survived

0 = Did not survive, 1 = Survived Pclass: Passenger ticket class where 1 = First class, 2 = Second class, 3 = Third class

Passenger ticket class where 1 = First class, 2 = Second class, 3 = Third class Sex: Male or female

Male or female Age: Age in years

Age in years SibSp: Number of siblings or spouses on the Titanic

Number of siblings or spouses on the Titanic Parch: Number of parents or children on the Titanic

Number of parents or children on the Titanic Ticket: Passenger ticket number

Passenger ticket number Fare: Passenger fare

Passenger fare Cabin: Passenger cabin number

Passenger cabin number Embarked: Point of embarkation where C = Cherbourg, Q = Queenstown, S = Southampton

Exploratory data analysis (EDA)

Exploratory data analysis is the process of visualising and analysing data to extract insights. We will mostly be using the Pandas library for this task. I have an extensive tutorial on Pandas which you can check out here.

In this section, I will discuss the key results of my EDA. Again, you can find the full analysis on my notebook.

I first split all the features in the dataset into categorical and numerical variables and analyse them individually to see how each correlate with survival.

When performing feature analysis, it is important to make the distinction between categorical and numerical variables because it will help us structure our analysis more properly. For instance, we cannot compute the average of a categorical variable such as gender because average can only be applied to a numerical variable which has a continuous distribution of values.

Categorical variables in the training set are Sex, Pclass and Embarked. Numerical variables, on the other hand, include SibSp, Parch, Age and Fare.

Below are some of the insights that I have gathered from the EDA process:

Female passengers are far more likely to survive than male passengers. If I remember correctly from the movie, women and children were prioritised during the evacuation of the Titanic so it makes sense that women have a higher chance of survival than men.

Female passengers were more likely to survive than male passengers

First-class passengers were more likely to survive than second class passengers and second class passengers were more likely to survive than third-class passengers. First-class passengers are individuals with high social status, influence and wealth. It wouldn’t surprise me if they were prioritised over the other passengers during evacuation.

First-class passengers had the highest survival probability; third-class passengers had the lowest survival probability

Fare is the most (positively) correlated numerical feature to survival. The more a passenger pays for their ticket, the more likely he/she is at surviving. Makes sense! Refer to the above dot point about first-class passengers.

Fare is the most correlated to survival at 0.26

Passengers of younger ages especially children have a higher survival probability than the other passengers. Again, “women and children first” during evacuation.

Passengers of younger ages especially children were more likely to survive

Data preprocessing

Data preprocessing is the process of getting our training set ready for model training. Here, we deal with missing values, apply data transformation, do feature engineering as well as label encoding.

I will now briefly go through each of these steps but I highly suggest you refer to my notebook to have a better understanding of what is being discussed here.

Missing values: There are two ways of dealing with missing values, by dropping or by filling. Filling missing values with substituted values is sometimes called imputation. Each method has its own individual trade-off between convenience and accuracy. Concretely, dropping missing values from our dataframe is the easiest and most convenient way to treat missing data. However, it often comes at the cost of losing potentially useful information in our dataset. On the flip side, imputation requires more time and consideration to effectively execute. Therefore, you should consider the state of the missing data in your dataset in order to decide on the most appropriate approach to deal with them. As a rule of thumb, if a column or feature is severely missing, we can safely drop it from the dataset. In relation to the Titanic competition, I decided to drop the Cabin feature and fill the Embarked, Fare and Age features.

There are two ways of dealing with missing values, by dropping or by filling. Filling missing values with substituted values is sometimes called imputation. Each method has its own individual trade-off between convenience and accuracy. Concretely, dropping missing values from our dataframe is the easiest and most convenient way to treat missing data. However, it often comes at the cost of losing potentially useful information in our dataset. On the flip side, imputation requires more time and consideration to effectively execute. Therefore, you should consider the state of the missing data in your dataset in order to decide on the most appropriate approach to deal with them. As a rule of thumb, if a column or feature is severely missing, we can safely drop it from the dataset. In relation to the Titanic competition, I decided to drop the Cabin feature and fill the Embarked, Fare and Age features. Data transformation: This step is specific to the Fare column in the dataset. The Fare distribution has a high positive skewness, that is the distribution leans towards the left-hand side. Skewness is when values are concentrated on one end of the distribution. This can have an adverse effect on our model’s ability to make accurate predictions. Hence, it is crucial that we address this issue via data transformation. More specifically, I have applied a log transformation to the Fare column and this resulted in a drastic decrease in skewness from 4.51 down to 0.57.

Passenger fare before log transformation

Passenger fare after log transformation

Feature engineering: Feature engineering is the process of creating new features from existing features to better represent the underlying problem to the predictive models. This is arguably the most important art in machine learning. The reason that it is called an art is because feature engineering often comes with experience and domain expertise, that is knowledge of a particular problem or industry. In my notebook, I created three new features and they are Title, IsAlone and Age*Class.

Feature engineering is the process of creating new features from existing features to better represent the underlying problem to the predictive models. This is arguably the most important art in machine learning. The reason that it is called an art is because feature engineering often comes with experience and domain expertise, that is knowledge of a particular problem or industry. In my notebook, I created three new features and they are Title, IsAlone and Age*Class. Label encoding: Machine learning models require all input and output variables to be numeric. So, we need to encode all categorical data before fitting them to our models.

Model training

Here comes the most exciting part of the competition, modelling!

Scikit-learn is one of the most popular libraries for machine learning if you are coding in the Python language. They also have a comprehensive documentation on the tools that are included in the library for data preprocessing, modelling, model evaluation and hyperparameter tuning.

Before we can fit the training set to our models, we need to first split the training set into predictor variables and response variable.

I have chosen to fit the training set to ten different classifiers and they are:

Logistic regression Support vector classifier K-nearest neighbours Gaussian naive bayes Perceptron Linear stochastic vector classifier Stochastic gradient descent Decision tree Random forest CatBoost

Modelling in Scikit-learn entails three simple steps. Firstly, we need to instantiate our model, that is simply declaring a model and assigning it to a variable. Next, we need to fit the model to our training set, both the predictor variables and the response variable. Finally, we can use this model to make predictions on the test set.

Model evaluation

An important concept in model evaluation is cross-validation. Recall a model that only fits well to training data but unable to make predictions on new data is basically useless. Cross-validation offers a way for us to test the accuracy of our model at predicting new data.

Cross-validation is the process whereby we repeatedly train our model with only a subset of our training set with the remaining data reserved for testing later on. The reserved data is sometimes called the holdout set.

Cross-validation provides a more accurate evaluation of model accuracy than merely training accuracy. This is because training accuracy overlooks the problem of overfitting. Overfitting is when our model learns the noise in our data rather than the signal. The idea behind a holdout set is so that our model can be assessed on its ability to predict on data that it was not trained on.

In my notebook, I went with support vector classifier out of the other models because it had the highest cross-validation mean.

Support vector classifier has the highest cross-validation mean

Hyperparameter tuning

Hyperparameter tuning is the process of tuning the parameters of the model. Using GridSearchCV, I managed to tune the parameters of my support vector classifier and saw a slight improvement in model accuracy.

Final results

Using my freshly tuned support vector classifier, I made predictions on the test set and managed to achieve a submission score of 0.77511 when submitted to Kaggle. In other words, I successfully predicted 77.5% of the passenger data in the test set.

Possible extensions to improve model accuracy

I did this project a few months ago and I haven’t really had the chance to revise it and improve my model accuracy. However, here I have several possible extensions that you can potentially add to your project to make it better than mine:

Analyse the ticket and cabin columns rather than dropping them

Come up with alternative features in feature engineering than what I have

Remove features that are less important to reduce overfitting

Try ensemble modelling which is combining the results from various machine learning classifiers

Conclusion

The Titanic survival prediction competition is an example of a classification problem in machine learning.

In this project, we analyse different features of the passengers aboard the Titanic and subsequently build a machine learning model that can classify the outcome of these passengers as either survived or did not survive.

After fitting ten different classifiers to my training data, support vector classifier showed the most promising and accurate prediction results. Therefore, I have chosen this classifier as my model of choice and achieved a submission score of 0.77511, that is I correctly predicted 77.5% of the passenger data in the test set.

I hope this article and my notebook help you get around the initial hump that I had when I first started learning about machine learning and more importantly, inspire you to participate in more Kaggle competitions in the future.

You can find the full notebook on my GitHub here. Feel free to reach out to me if you have any questions.

Happy learning!

Video Tutorial

If you prefer learning via video, I have two videos up on my YouTube channel which walks through this project in more detail. Do check them out if you are interested!

Part 1: Exploratory data analysis