Predicting survivors of Titanic. Who will survive a shipwreck? We can... | by Dorian Lazar | Towards Data Science

RMS Titanic was a British passenger liner operated by the White Star Line that sank in the North Atlantic Ocean in the early morning hours of April 15, 1912, after striking an iceberg during her maiden voyage from Southampton to New York City. Of the estimated 2,224 passengers and crew aboard, more than 1,500 died, making the sinking one of modern history’s deadliest peacetime commercial marine disasters.
What we want now is to create a machine learning model that is able to predict who would survive Titanic’s shipwreck. For that we will use this dataset from Kaggle — there is also a Kaggle competition for this task which is a good place to get started with Kaggle competitions. This dataset contains the following information about the passengers:
Now, before doing any machine learning on this dataset, it is a good practice to do some exploratory analysis and see how our data looks like. We also want to prepare/clean it along the way.
import numpy as npimport pandas as pddf = pd.read_csv('train.csv')df
df.describe()
What do we see in our data at first glance? We see that the PassengerId column has an unique number associated with each passenger. This field can easily be used by machine learning algorithms to just memorize the outcomes for each passenger without the ability to generalize. Also, there is the Name variable which I personally don’t think it determines in any way if one person survived or not. So, we will delete these two variables.
del df['PassengerId']del df['Name']
Now, let’s see if we have missing values and how many they are.
df.isnull().sum()
Out of 891 samples, 687 have null values for the Cabin variable. There are too many missing values for this variable to be able to make use of it. We will delete it.
del df['Cabin']
We don’t want to delete the other two columns as they have few missing values, we will augment them.For the Embarked variable, as it is a categorical variable, we will look at the counts of each category and replace the 2 null values with the category that has most items.
df['Embarked'].value_counts()
The most frequent value for Embarked is ‘S’, so we will use it to replace the null values.
df['Embarked'].loc[pd.isnull(df['Embarked'])] = 'S'
For age we will replace missing values with the average age.
mean_age_train = np.mean(df['Age'].loc[pd.isnull(df['Age']) == False].values)df['Age'].loc[pd.isnull(df['Age'])] = mean_age_train
Note that we should store everything that we learn from our training data, such as Embarked class frequency or Age mean, as we will use this information when making predictions in case there are also missing values in test data.We will also compute and store Fare mean in case we need it on test data.
mean_fare_train = np.mean(df['Fare'].loc[pd.isnull(df['Fare']) == False].values)
We got rid of null values, now let’s see what’s next.
Machine learning algorithms don’t work with text (at least not directly), we need to convert all strings with numbers. We will use ordinal encoding for this conversion. Ordinal encoding is a way to convert a categorical variable into numbers by assigning each category a number. We’re going to use Scikit-Learn’s OrdinalEncoder to apply this transformation on variables Sex, Ticket, Embarked.Before that, we will backup our current data format for future use.
df_bkp = df.copy()from sklearn.preprocessing import OrdinalEncoderdf['Sex'] = OrdinalEncoder().fit_transform(df['Sex'].values.reshape((-1, 1)))df['Ticket'] = OrdinalEncoder().fit_transform(df['Ticket'].values.reshape((-1, 1)))df['Embarked'] = OrdinalEncoder().fit_transform(df['Embarked'].values.reshape((-1, 1)))
Now we will visualize our data to see how the distribution of our variables varies across survived and not-survived classes.Below are histograms for each variable in our dataset (along with the code that generated it), in the left is the subset of people who survived, and in the right who didn’t survive.
import matplotlib.pyplot as pltfrom IPython.display import display, Markdowndef show(txt):    # this function is for printing markdown in jupyter notebook    display(Markdown(txt))for i in range(1, 9):    show(f'### {df.columns[i]}')    f, (survived, not_survived) = plt.subplots(1, 2, sharey=True, figsize=(18, 8))    survived.hist(df.iloc[np.where(df['Survived'] == 1)[0], i])    survived.set_title('Survived')not_survived.hist(df.iloc[np.where(df['Survived'] == 0)[0], i])    not_survived.set_title('Not Survived')    plt.show()
Now, we will try a couple of machine learning methods on our dataset to see what results we get. The machine learning methods that we will use are:
Logistic Regression
Support Vector Machine
Decision Tree
K Nearest Neighbors
Multi-Layer Perceptron
Instead of choosing a fixed validation set to estimate the accuracy of the test set, we will use (5-fold) cross validation with Scikit-Learn’s cross_val_score which returns an array with scores for each cross validation iteration.
from sklearn.model_selection import cross_val_scorefrom sklearn.linear_model import LogisticRegressionfrom sklearn.svm import SVCfrom sklearn.tree import DecisionTreeClassifierfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.neural_network import MLPClassifierX = df.iloc[:, 1:].valuesy = df.iloc[:, 0].values# Logistic Regressionlr = LogisticRegression()lr_score = np.mean(cross_val_score(lr, X, y))print(f'Logistic Regression: {lr_score}')# Support Vector Machinesvc = SVC()svc_score = np.mean(cross_val_score(svc, X, y))print(f'Support Vector Machine: {svc_score}')# Decision Treedtc = DecisionTreeClassifier()dtc_score = np.mean(cross_val_score(dtc, X, y))print(f'Decision Tree: {dtc_score}')# K Nearest Neighborsknc = KNeighborsClassifier()knc_score = np.mean(cross_val_score(knc, X, y))print(f'K Nearest Neighbors: {knc_score}')# Multi-Layer Perceptronmlpc = MLPClassifier()mlpc_score = np.mean(cross_val_score(mlpc, X, y))print(f'Multi-Layer Perceptron: {mlpc_score}')
Running this code we got the following results:
Let’s see if we can improve the accuracy by working on our dataset’s features. For that, we will restore first to the backup we made before applying ordinal encoding.
df = df_bkp.copy()
Instead of ordinal encoding, now we want to apply one-hot encoding to our categorical variables. With one-hot encoding, instead of assigning each class a number, we assign a vector with all 0’s except for one position specific to that class in which we put a 1. That is, we transform each class into a variable on its own that takes the value 0 or 1. We will apply one-hot encoding on the variables Pclass, Sex, Ticket and Embarked using Scikit-Learn’s OneHotEncoder.
from sklearn.preprocessing import OneHotEncoder# Pclasspclass_transf = OneHotEncoder(sparse=False, dtype=np.uint8, handle_unknown='ignore')pclass_transf.fit(df['Pclass'].values.reshape((-1, 1)))pclass = pclass_transf.transform(df['Pclass'].values.reshape((-1, 1)))df['Pclass0'] = pclass[:, 0]df['Pclass1'] = pclass[:, 1]df['Pclass2'] = pclass[:, 2]del df['Pclass']# Sexgender_transf = OneHotEncoder(sparse=False, dtype=np.uint8, handle_unknown='ignore')gender_transf.fit(df['Sex'].values.reshape((-1, 1)))gender = gender_transf.transform(df['Sex'].values.reshape((-1, 1)))df['Male'] = gender[:, 0]df['Female'] = gender[:, 1]del df['Sex']# Ticketticket_transf = OneHotEncoder(sparse=False, dtype=np.uint8, handle_unknown='ignore')ticket_transf.fit(df['Ticket'].values.reshape((-1, 1)))ticket = ticket_transf.transform(df['Ticket'].values.reshape((-1, 1)))for i in range(ticket.shape[1]):    df[f'Ticket{i}'] = ticket[:, i]del df['Ticket']# Embarkedembarked_transf = OneHotEncoder(sparse=False, dtype=np.uint8, handle_unknown='ignore')embarked_transf.fit(df['Embarked'].values.reshape((-1, 1)))embarked = embarked_transf.transform(df['Embarked'].values.reshape((-1, 1)))for i in range(embarked.shape[1]):    df[f'Embarked{i}'] = embarked[:, i]del df['Embarked']
We also want to scale the numerical variables to [0, 1] range. For that we will use MinMaxScaler which scales the variables so that the minimum value is moved to 0, the maximum to 1 and other intermediary values are scaled accordingly in between 0, 1. We will apply this transformation to variables Age, SibSp, Parch, Fare.
from sklearn.preprocessing import MinMaxScalerage_transf = MinMaxScaler().fit(df['Age'].values.reshape(-1, 1))df['Age'] = age_transf.transform(df['Age'].values.reshape(-1, 1))sibsp_transf = MinMaxScaler().fit(df['SibSp'].values.reshape(-1, 1))df['SibSp'] = sibsp_transf.transform(df['SibSp'].values.reshape(-1, 1))parch_transf = MinMaxScaler().fit(df['Parch'].values.reshape(-1, 1))df['Parch'] = parch_transf.transform(df['Parch'].values.reshape(-1, 1))fare_transf = MinMaxScaler().fit(df['Fare'].values.reshape(-1, 1))df['Fare'] = fare_transf.transform(df['Fare'].values.reshape(-1, 1))
Now we will run the same machine learning algorithms on this new data format to see what results we get.
X = df.iloc[:, 1:].valuesy = df.iloc[:, 0].values# Logistic Regressionlr = LogisticRegression()lr_score = np.mean(cross_val_score(lr, X, y))print(f'Logistic Regression: {lr_score}')# Support Vector Machinesvc = SVC()svc_score = np.mean(cross_val_score(svc, X, y))print(f'Support Vector Machine: {svc_score}')# Decision Treedtc = DecisionTreeClassifier()dtc_score = np.mean(cross_val_score(dtc, X, y))print(f'Decision Tree: {dtc_score}')# K Nearest Neighborsknc = KNeighborsClassifier()knc_score = np.mean(cross_val_score(knc, X, y))print(f'K Nearest Neighbors: {knc_score}')# Multi-Layer Perceptronmlpc = MLPClassifier()mlpc_score = np.mean(cross_val_score(mlpc, X, y))print(f'Multi-Layer Perceptron: {mlpc_score}')
After we engineered our features in this way we got an significant improvement in the accuracy of all classifiers. The best classifier among them being Decision Tree Classifier. Now we try to improve it by doing hyper-parameter tuning using GridSearchCV.
from sklearn.model_selection import GridSearchCVdtc = DecisionTreeClassifier()params = {    'max_depth': list(range(2, 151)),    'min_samples_split': list(range(2, 15))}clf = GridSearchCV(dtc, params)clf.fit(X, y)print(f'Best params: {clf.best_params_}')print(f'Best score: {clf.best_score_}')
The parameters that we got are:
And the score that we got is 84.40%, an improvement of 0.9% after hyperparameter tuning.
I hope you found this information useful and thanks for reading!
This article is also posted on my own website here. Feel free to have a look!