Machine Learning to Predict the Survivals of Titanic
By: Kheirie Elhariri
Tags: Machine Learning, Data Science, Xgboost, Beginner Project, Titanic

1. Necessary Libraries

The libraries/packages that are used in this projects are:

pandas: a library used for data analysis and processing.

matplotlib: a library used for data visualization. It is a plotting library for Python.

seaborn: a library built on top of matplotlib. It is also used for data visualization.

scikit-learn: a machine learning library that supports various supervised and unsupervised learning algorithms.

XGBoost: a library providing gradient boosted decision trees. It is well known for its high performance and speed.

Interesting fact :) pandas’ name is a play on words for "Python Data Analysis". According to wikipedia, the name is derived from "panel data", a term used in econometrics to describe datasets that include observations over multiple time periods for the same individuals.

import pandas as pd

import matplotlib.pypolt as plt

import seaborn as sns from sklearn.model_selection import train_test_split

from xgboost import XGBClassifier

from sklearn.model_selection import RandomizedSearchCV

2. Acquiring Data

The first step in any data science project, is to acquire data. Data can come in many forms. In this project the train and test data are structured labelled data imported from CSV files.

train_data = pd.read_csv("train.csv")

test_data = pd.read_csv("test.csv")

3. Exploring and Preprocessing Data

In this section data is analyzed by the help of data visualization and preprocessed depending on the analysis results. Note that all data preprocessing that is done on the train data should also be done on the test data.

train_data.info() in Python helps to get a brief summary of the dataset

train_data.info()

columns = train_data.columns for col in columns:

print("unique values in {} column is: {}". format(col,

train_data[col].value_counts().size))

____________________________________________________________________ unique values in PassengerId column is: 891

unique values in Survived column is: 2

unique values in Pclass column is: 3

unique values in Name column is: 891

unique values in Sex column is: 2

unique values in Age column is: 88

unique values in SibSp column is: 7

unique values in Parch column is: 7

unique values in Ticket column is: 681

unique values in Fare column is: 248

unique values in Cabin column is: 147

unique values in Embarked column is: 3

____________________________________________________________________

The training set consists of 891 rows. Each passenger has a unique PassengerId. The summary shows that some features have missing values, and that there are three different types of features, int64, float64 and object.

train_data[col].value_counts().size permits to understand the number of unique values/elements of each column (i.e.: feature). The pandas.series.value_counts returns the size of unique values in a series.

Sections 3.1–3.8 describe the features and discuss how they were dealt with.

3.1 Sex

The Sex feature is used to get an overview of the distribution of sexes on the ship.

print("Total number of male passengers:")

print(train_data.loc[train_data.Sex=="male"].Sex.size)

print("Total number of female passengers:")

print(train_data.loc[train_data.Sex=="female"].Sex.size)

____________________________________________________________________ Total number of male passengers: 577

Total number of female passengers: 314

____________________________________________________________________

Even though there are more male passengers than female, the data reveals that women had more survival chance that men.

print("The percentage of survived with respect to Sex:")

print(100 * train_data.groupby("Sex").Survived.mean())

____________________________________________________________________ The percentage of survived with respect to Sex:

Sex

female 74.203822

male 18.890815

Name: Survived, dtype: float64

____________________________________________________________________

3.2 Age

Since Age is of type float64, a histogram is used to understand the relation between Survived and Age features.

g = sns.FacetGrid(col="Survived", data=train_data, height = 2, aspect=3)

g.map(sns.distplot, "Age", kde=False, bins=80)

The histograms reveal that children had a higher survival chance. For a better understanding, Age feature is divided into categories by using pandas.cut.

def cut_age(df, cut_values, label_names): # fill missing values with -0.5

df["Age"] = df["Age"].fillna(-0.5)

df["Age"]=pd.cut(df["Age"], bins=cut_values, labels=label_names)

return df cut_values=[-1, 0, 3, 12, 19, 35, 60, 80] label_names=["Missing", "Infants", "Children", "Teenagers", "Young Adults", "Middle-Age Adults", "Seniors"] train_data=cut_age(train_data, cut_values, label_names)

test_data=cut_age(test_data, cut_values, label_names) sns.catplot(x="Age", row="Survived", kind="count", height=3, aspect=4, data=train_data)

print(100 * train_data.groupby("Age").Survived.mean())

____________________________________________________________________ Age

Missing 29.378531

Infants 66.666667

Children 51.282051

Teenagers 41.052632

Young Adults 38.438438

Middle-Age Adults 40.000000

Seniors 22.727273

Name: Survived, dtype: float64

____________________________________________________________________

The results do prove that children survived more than other age groups.

3.3 SibSp and Parch

SibSp feature indicates a passenger's relation with siblings (i.e.: brother/sister) and spouse (i.e.: husband/wife)

Parch feature indicates a passenger's relation with parents (i.e.: father/mother) or children on the ship

print("The percentage of survived with respect to SibSp:")

print(100 * train_data.groupby("Age").Survived.mean())

____________________________________________________________________ The percentage of survived with respect to SibSp:

SibSp

0 34.539474

1 53.588517

2 46.428571

3 25.000000

4 16.666667

5 0.000000

8 0.000000

Name: Survived, dtype: float64

____________________________________________________________________ print("The percentage of survived with respect to Parch:")

print(100 * train_data.groupby("Parch").Survived.mean())

____________________________________________________________________ The percentage of survived with respect to Parch:

Parch

0 34.365782

1 55.084746

2 50.000000

3 60.000000

4 0.000000

5 20.000000

6 0.000000

Name: Survived, dtype: float64

____________________________________________________________________

The results reveal that individuals who are alone (no family members) had less chance of survival than if they had 1 to 3 family members on board. But, when the family members are more than 3 the survival chance drops. To be more precise, Parch and SibSp features will be combined into one feature called Fam_membs.

def Cr_fam_membs(df):

df["FamMembs"]= df["Parch"] + df["SibSp"]

df=df.drop(["SibSp", "Parch"], axis=1)

return df train_data=Cr_fam_membs(train_data)

test_data=Cr_fam_membs(test_data) print(100 * train_data.groupby("FamMembs").Survived.mean())

____________________________________________________________________

FamMembs

0 30.353818

1 55.279503

2 57.843137

3 72.413793

4 20.000000

5 13.636364

6 33.333333

7 0.000000

10 0.000000

Name: Survived, dtype: float64

____________________________________________________________________

To limit the number of categories in Fam_membs features, it will be divided into 4 categories as following:

IsAlone: 0 fam_membs

Small family: 1–3 fam_membs

Medium family: 4–6 fam_membs

Large family: 7–10 fam_membs

train_data["FamMembs"]=train_data["FamMembs"].apply(lambda s: "IsAlone" if s==0 else s)

train_data["FamMembs"]=train_data["FamMembs"].apply(lambda s: "Small family" if (s==1 or s==2 or s==3) else s)

train_data["FamMembs"]=train_data["FamMembs"].apply(lambda s: "Meduim family" if (s==4 or s==5 or s==6) else s)

train_data["FamMembs"]=train_data["FamMembs"].apply(lambda s: "Large family" if (s==7 or s==10) else s) test_data["FamMembs"].value_counts()

____________________________________________________________________



IsAlone 253

Small family 145

Meduim family 14

Large family 6

Name: FamMembs, dtype: int64

____________________________________________________________________

FamMember feature makes it easier to understand how the size of family effects the survival chance.

print("The percentage of survived with respect to Fam_membs:")

print(100 * train_data.groupby("FamMembs").Survived.mean())

____________________________________________________________________The percentage of survived with respect to Fam_membs:

FamMembs

IsAlone 30.353818

Large family 0.000000

Meduim family 20.408163

Small family 57.876712

Name: Survived, dtype: float64

____________________________________________________________________ sns.catplot(x="FamMembs", row="Survived", kind="count", height=3, aspect=4, data=train_data)

3.4 Pclass

The Pclass represents the passenger's class. It is divided into three classes 1st, 2nd and 3rd. As expected, passengers belonging to the 1st class had the highest rate of survival and passengers belonging to the 3rd class had the lowest rate.

print(“The percentage of survived with respect to Pclass:”)

print(100 * train_data.groupby(“Pclass”).Survived.mean())

____________________________________________________________________The percentage of survived with respect to Pclass:

Pclass

1 62.962963

2 47.282609

3 24.236253

Name: Survived, dtype: float64

____________________________________________________________________

3.5 Embarked

The ports of embarkations are C = Cherbourg, Q = Queenstown and S = Southampton. Since the missing values in the Embarked feature are only limited to 2, in the training data, they are filled by the most frequent values.

train_data["Embarked"]=train_data["Embarked"].fillna(train_data["Embarked"].mode()[0])

3.6 Fare

Similar to the Age feature, the Fare is divided into categories using pandas.qcut. The main difference between pandas.qcut and pandas.cut is that pandas.qcut will create equal sized bins, whereas pandas.cut is used to exactly specify the edges of the bins. pandas.cut is not used since it is not obvious how the Fare was assigned and how it relates to other features; thus, the edges of the bins can not be easily determined. Check this notebook for more details about how the Fare feature is analyzed with respect to Pclass and Embarked features.

def qcut_fare(df, q, labels):

df["Fare"]=pd.qcut(df["Fare"], q, labels=labels)

return df labels=["range1", "range2", "range3", "range4"]

train_data=qcut_fare(train_data, 4, labels)

q=4 is used to divide the data according to the default percentiles of pandas.Series.describe (i.e.: range1: min-0.25, range2: 0.25-0.5, range3: 0.5-0.75, range4: 0.75-max).

train_data[“Fare”].describe()

____________________________________________________________________count 891.000000

mean 32.204208

std 49.693429

min 0.000000

25% 7.910400

50% 14.454200

75% 31.000000

max 512.329200

Name: Fare, dtype: float64

____________________________________________________________________ sns.catplot(x="Fare", data=train_data, kind="count", height=2, aspect=3)

3.7 Name

It is very easy to assume that the Name feature is not useful, since every individual has a unique name. However, looking closely at this feature, one can notice that names have titles, which can be useful for the prediction.

train_data["Name"]

____________________________________________________________________0 Braund, Mr. Owen Harris

1 Cumings, Mrs. John Bradley (Florence Briggs Th...

2 Heikkinen, Miss. Laina

3 Futrelle, Mrs. Jacques Heath (Lily May Peel)

4 Allen, Mr. William Henry

...

886 Montvila, Rev. Juozas

887 Graham, Miss. Margaret Edith

888 Johnston, Miss. Catherine Helen "Carrie"

889 Behr, Mr. Karl Howell

890 Dooley, Mr. Patrick

Name: Name, Length: 891, dtype: object

____________________________________________________________________

The names in the name feature are replaced by titles (i.e.: Mr., Mrs., Miss), if available, or 'Other' (in the cases where titles are not available or very rare).

train_data["Name"]=train_data["Name"].apply(lambda s: s.split(', ')[1].split('.')[0]) train_data["Name"].value_counts()

____________________________________________________________________Mr 517

Miss 182

Mrs 125

Master 40

Dr 7

Rev 6

Major 2

Col 2

Mlle 2

Mme 1

Sir 1

Ms 1

the Countess 1

Capt 1

Jonkheer 1

Don 1

Lady 1

Name: Name, dtype: int64

____________________________________________________________________ train_data["Name"]=train_data["Name"].replace(["Ms", "Mlle"], "Miss")

train_data["Name"]=train_data["Name"].replace(["Sir"], "Mr")

train_data["Name"]=train_data["Name"].replace(["Mme"], "Mrs")

train_data["Name"]=train_data["Name"].replace(["Dr", "Rev", "Col", "Major", "Capt", "Master", "Lady", "the Countess", "Don", "Dona", "Jonkheer"], "Other") sns.catplot(x="Name", hue="Survived", kind="count", data=train_data)

As deduced from the Sex Feature, the above graph proves that women ('Mrs. and 'Miss') had more survival chance than men ('Mr').

3.8 Cabin

Even though the Cabin feature has a lot of missing values, it is not directly dropped. It is used to create a new feature called 'Deck'. Cabins are located on different decks, which are: A, B, C, D, E, F, G and T.

# fill the missing values with "Unknown"

train_data[“Cabin”]=train_data[“Cabin”].fillna(“Unknown”) # remove digits to create the deck feature

train_data["Deck"]=train_data["Cabin"].str.replace("([0-9\s])+","") train_data["Deck"].value_counts()

____________________________________________________________________Unknown 687

C 51

B 36

D 32

E 32

A 15

F 9

BB 7

CC 4

G 4

CCC 4

FG 3

BBB 2

BBBB 2

FE 1

T 1

DD 1

Name: Deck, dtype: int64

____________________________________________________________________

The data reveals that some passengers had more than one cabin. Some of which were on the same deck and others on different decks. To deal with this, a new feature is created to indicate the total number of cabins per passenger. If the passenger’s cabin is unknown it is 0. For decks which constitute of more than one letter, the first letter is taken.

def total_cabins(row):

if row.Deck == "Unknown":

row["TotalCab"] = 0

elif len(row.Deck) > 1:

row["TotalCab"] = len(row.Deck)

else:

row["TotalCab"] = 1

return row train_data=train_data.apply(total_cabins, axis=1) train_data["TotalCab"].value_counts()

____________________________________________________________________0 687

1 180

2 16

3 6

4 2

Name: TotalCab, dtype: int64

____________________________________________________________________ train_data["Deck"]=train_data["Deck"].apply(lambda s: s[0] if s != "Unknown" else s) train_data["Deck"].value_counts()

____________________________________________________________________Unknown 687

C 59

B 47

D 33

E 32

A 15

F 13

G 4

T 1

Name: Deck, dtype: int64

____________________________________________________________________

3.9 Drop unnecessary columns

train_data=train_data.drop(["Survived", "Cabin", "Ticket"], axis=1)

4. Dealing with Categorical Values

Machine learning models only understand numerical values. One-hot-encoding is used to transform non-numeric data to numeric data.

from sklearn.preprocessing import OneHotEncoder OHE = OneHotEncoder(handle_unknown='ignore', sparse=False) features = ["Pclass", "Name", "Sex", "Age", "Fare", "Embarked",

"FamMembs", "Deck", "TotalCab"] OHE_train_cols=pd.DataFrame(OHE.fit_transform(train_data[features])) OHE_train_cols.index=train_data.index



num_train=train_data.drop(features, axis=1) train_data = pd.concat([num_train, OHE_train_cols], axis=1) print(train_data.shape)

____________________________________________________________________ (891, 42)

____________________________________________________________________

5. XGBoost Parameter Tuning using RandomizedSearchCV

XGBoost stands for Extreme Gradient Boosting. It is a decision tree based model ensemble algorithm that implements gradient boosting framework. It is designed for speed and performance. XGBoost proved its superiority when dealing with structured/tabular data, which is the case of this project.

The problem that we are solving in this project is considered as a classification problem, since the outcome is either 1 (survived) or 0 (did not survive). XGBClassifier is used in this case.

RandomizedSearchCV is used to find the best parameters. It passes the sets of given parameters, calculates the score and gives the best set of parameters. This process is sometimes called as hyper-parameter tuning. It also performs train_test_split.

from xgboost import XGBClassifier

from sklearn.model_selection import RandomizedSearchCV xgb=XGBClassifier(objective='reg:logistic') params={

'n_estimators': [200, 500, 1000],

'learning_rate': [0.01, 0.05, 0.1],

'max_depth': [5, 7, 9],

'colsample_bytree': [ 0.4, 0.6, 0.8],

'subsample': [0.8, 0.9, 1],

'gamma': [0, 0.5, 1]

} # split the data into train and validation data

X_train, X_valid, y_train, y_valid = train_test_split(train_data, y, test_size=0.25) clf=RandomizedSearchCV(xgb, param_distributions=params, n_iter=50, n_jobs=-1, verbose=1) clf.fit(train_data, y)



score=clf.best_score_

params=clf.best_params_

print("Best score: ",score)

print("Best parameters: ", params) ____________________________________________________________________ Best score: 0.812874251497006

Best parameters: {'subsample': 1, 'n_estimators': 500, 'max_depth': 9, 'learning_rate': 0.01, 'gamma': 0.5, 'colsample_bytree': 0.4}

____________________________________________________________________

In order to find the accuracy, the predictions are compared to the validation target.