Building a Machine Learning Model Step By Step With the Titanic Dataset
By: Taha Bilal Uyar
Tags: Machine Learning, Data Science, Kaggle, Pandas, Data Visualization

Feature Engineering and Modelling

We will start with random forest classification model for this data set. Its nonlinear nature makes it usually a great option for not only classification but also regression problems.

We will start with numerical features to create our machine learning model at the beginning.

X=train_data[["Pclass", "Age", "SibSp", "Parch", "Fare"]]

y=train_data["Survived"]

We split our data set into two parts which are train and test sets. It is necessary to check the model’s accuracy. We cannot be aware of a possible overfitting problem by using only one data.

from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

After splitting our data set into train and test tests, we create our model by using sklearn.RandomForestClassifier(). Then we fit our model to the train data then make prediction for test data.

from sklearn.ensemble import RandomForestClassifier rfc=RandomForestClassifier()

rfc.fit(X_train, y_train)

rfc.score(X_test,y_test)

It doesn’t work as expected because we need to complete missing values before the deploying of model.

train_data.info()

test_data.info()

As we can see, there missing values in some features which are Age, Cabin, Fare, and Embarked. Firstly, we will get rid of NaN values in numerical features.

train_data["Age"].fillna(train_data["Age"].mean(), inplace=True)

test_data["Age"].fillna(test_data["Age"].mean(), inplace=True)

test_data["Fare"].fillna(test_data["Fare"].mean(), inplace=True)

NaN values are replaced with mean of the column by using fillna method of pandas in both train and test data.

Now, we can deploy our model.

X=train_data[["Pclass", "Age", "SibSp", "Parch", "Fare"]]

y=train_data["Survived"] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) rfc=RandomForestClassifier(random_state=35)

rfc.fit(X_train, y_train) print("test accuracy: ",rfc.score(X_test,y_test))

We could reach 68% accuracy by using only numerical features. We will try to increase our accuracy. We have some other features whose data type is an object. These features also have some correlation with surviving that we already saw in the previous part.

We have to convert our string values to numerical values. There are 2 main methods here. The first method is one hot encoding which creates a column for each unique value and fills these columns with binary codes 1 and 0. The other one is label encoding that replaces each unique string value with a unique number.

Both pandas and sckit learn have proper functions for this process. One hot encoding is possible with pandas.get_dummies and sklearn.preprocessing.OneHotEncoder. For label encoding, we can use pandas.factorize and sklearn.preprocessing.LabelEncoder functions.

train_data["Sex_encoded"]=pd.factorize(train_data["Sex"])[0]

train_data["Embarked_encoded"]=pd.factorize(train_data["Embarked"])[0] test_data["Sex_encoded"]=pd.factorize(test_data["Sex"])[0]

test_data["Embarked_encoded"]=pd.factorize(test_data["Embarked"])[0]

We do not need to replace null values in the embarked column in train data. Because pandas.factorize function encodes null values separately.

train_data["Embarked"].unique()

train_data["Embarked_encoded"].unique()

NaN values replaced with -1 as you can see above.

Now, we can try one more time to deploy our model with new features.

X=train_data[["Pclass", "Age", "SibSp", "Parch", "Fare", "Sex_encoded", "Embarked_encoded"]] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) rfc=RandomForestClassifier(random_state=35)

rfc.fit(X_train, y_train) print("test accuracy: ",rfc.score(X_test,y_test))

As we can see, there is an increase in accuracy which is almost 10%.

We can add one more feature to our data set, which is the cabin column. However, looks like it is not possible by using panda.factorize function directly because there are too many different values. Firstly, we will try to acquire only deck codes.

Firstly, we complete null values then we take only the first letter of each row by using str.slice function from pandas. In this way, we can acquire cabin codes.

train_data["Cabin"].unique()

train_data["Cabin"].fillna("N", inplace=True)

train_data['Cabin_code'] = train_data["Cabin"].str.slice(0,1)

train_data['Cabin_code'].unique()

test_data["Cabin"].fillna("N", inplace=True)

test_data['Cabin_code'] = test_data["Cabin"].str.slice(0,1)

We can use also this feature to train our model anymore.

X=train_data[["Pclass", "Age", "SibSp", "Parch", "Fare", "Sex_encoded", "Embarked_encoded", "Cabin_code_encoded"]] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) rfc=RandomForestClassifier(random_state=35)

rfc.fit(X_train, y_train) print("test accuracy: ",rfc.score(X_test,y_test))

Adding this new feature to our X data set did not change the accuracy.

Random forest is a tree-based model so it doesn’t require feature scaling. Even it can soften the non-linear nature of the model. It is expected that result will be same after feature scaling. We can try and see.

from sklearn.preprocessing import StandardScaler scaler=StandardScaler()

X_train_sc=scaler.fit_transform(X_train)

X_test_sc=scaler.transform(X_test) rfc=RandomForestClassifier(random_state=35)

rfc.fit(X_train_sc, y_train)

print("train accuracy: ",rfc.score(X_train_sc, y_train))

print("test accuracy: ",rfc.score(X_test_sc,y_test))

As you can see, the test accuracy is the same before and after feature scaling (79.1%). So, standardization of features did not help to increase accuracy as we expected.

Overfitting

When we look at the train and test accuracies, we can see that the difference between train accuracy is not too small. I think it can not be called overfitting but still, we can try to decrease the gap between accuracies.

If we try to fit our model to each feature column, it can give us an insight into which columns cause this slight overfitting.

rfc=RandomForestClassifier(random_state=35)

for x in X_train.columns:



rfc.fit(X_train[[x]], y_train)

print(x,"train accuracy: ",rfc.score(X_train[[x]], y_train)*100)

print(x,"test accuracy: ",rfc.score(X_test[[x]],y_test)*100)

We can make some comments on these results. The sex column is the most important feature for our model, only itself has 79% test accuracy. And it also has almost the same train and test accuracy which is a very ideal situation. While the difference between train and test accuracies is very little for some features, especially numerical features such as age and fare have a bigger difference which is between 6–10%.

We can convert the Age and Fare features from numerical to label encoded to increase the efficiency of the model.

Age

train_data["Age"].groupby(train_data["Survived"]).plot(kind="hist", bins=20, legend=True, alpha=0.5)

train_data.loc[train_data['Age'] <= 7.5, 'Age_encoded'] = 0

train_data.loc[(train_data['Age'] > 7.5) & (train_data['Age'] <= 15), 'Age_encoded'] = 1

train_data.loc[(train_data['Age'] > 15) & (train_data['Age'] <= 25), 'Age_encoded'] = 2

train_data.loc[(train_data['Age'] > 25) & (train_data['Age'] <= 30), 'Age_encoded'] = 3

train_data.loc[(train_data['Age'] > 30) & (train_data['Age'] <= 35), 'Age_encoded'] = 4

train_data.loc[(train_data['Age'] > 35) & (train_data['Age'] <= 50), 'Age_encoded'] = 5

train_data.loc[train_data['Age'] > 50, 'Age_encoded'] = 6 train_data["Age_encoded"].unique()

train_data[['Age_encoded', 'Survived']].groupby(['Age_encoded'], as_index=False).mean()

sns.countplot(train_data["Survived"], hue=train_data["Age_encoded"])

Fare

train_data["Fare"].groupby(train_data["Survived"]).plot(kind="hist", bins=20, legend=True, alpha=0.5)

train_data.loc[train_data['Fare'] <= 12.5, 'Fare_encoded'] = 0

train_data.loc[(train_data['Fare'] > 12.5) & (train_data['Fare'] <= 25), 'Fare_encoded'] = 1

train_data.loc[(train_data['Fare'] > 25) & (train_data['Fare'] <= 50), 'Fare_encoded'] = 2

train_data.loc[(train_data['Fare'] > 50) & (train_data['Fare'] <= 75), 'Fare_encoded'] = 3

train_data.loc[(train_data['Fare'] > 75) & (train_data['Fare'] <= 100), 'Fare_encoded'] = 4

train_data.loc[(train_data['Fare'] > 100) & (train_data['Fare'] <= 150), 'Fare_encoded'] = 5

train_data.loc[train_data['Fare'] > 150, 'Fare_encoded'] = 6 train_data["Fare_encoded"].unique()

train_data[['Fare_encoded', 'Survived']].groupby(['Fare_encoded'], as_index=False).mean()

sns.countplot(train_data["Survived"], hue=train_data["Fare_encoded"])

As you can see from the above, Age and Fare columns were converted to discrete values from continuous ones. I tried to determine critical points and cut the features from those points.

I did it manually but there some functions for it in pandas. These are pandas.cut() and pandas.qcut() functions. They cut the values according to values and frequencies respectively.

Now, we can try our model with new features.

X=train_data[["Pclass", "Age_encoded", "SibSp", "Parch", "Fare_encoded", "Sex_encoded", "Embarked_encoded", "Cabin_code_encoded"]] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) rfc=RandomForestClassifier(random_state=35)

rfc.fit(X_train, y_train)

print("train accuracy: ",rfc.score(X_train, y_train))

print("test accuracy: ",rfc.score(X_test,y_test))

Our previous train and test accuracies were 98.2% and 79.1 respectively. Now train accuracy is 92.0% and test accuracy is 78.7%.

Converting the features to discrete values decreased the gap between train and test accuracies however, did not help to increase the test accuracy.