Machine Learning and Data Analysis with Python, Titanic Dataset: Part 4
By: Quinn Wang
Tags: Artificial Intelligence, Feature Engineering, Model Tuning, Machine Learning Tutorial, Machine Learning

Machine Learning and Data Analysis with Python, Titanic Dataset: Part 4

We have made our first submission to Kaggle in Part 3. In the final part of this series we are going to try to make improvements on our predictions and resubmit to see our results. Link to a video version of this tutorial will be at the bottom.

Let’s get started!

When it comes to improving our predictions, there are generally 2 things we want to consider:

Further feature engineering

Model tuning

But before we go any further, let’s take a look back at what was our submission score using the baseline model:

Baseline model score

You may have noticed some discrepancy here. Why is it that the test set from our 80–20 split scored an 82.68% accuracy, while predictions on the Kaggle test set only gave 76% accuracy?

One of the reasons is that there are some randomness in the ways these models predict. The 80% data we trained on might just happen to look more similar to the 20% we left as the simulated test set, compared to the test set Kaggle provided. And as a result the model trained from this 80% data would perform better on the 20% simulated test set. Especially when our training set is small, it is very likely for predictions to perform slightly better on one randomly selected test set than another.

Let me show you what I mean by running some tests on a different random selection of the 20% test set.

We were using random state 42 in our train test split function:

This number essentially determines which 20% of data gets picked so every time we run this line it’s going to use the same split. This is called a random seed, and it is useful in the sense that we get to have some control over the randomness in order to get fair comparisons between different trials.

Let’s see what happens when we change this random state to 0:

Results with random state 0 at train_test_split

Compare this with the result when random state is 42:

Results with random state 42 at train_test_split

Our accuracy score drop by over 2%.

Therefore, before we try to create more features or improve the model, in order to compare accuracies between any 2 models, we might want to take some sort of average to minimize the effect of this randomness.

To get the mean score:

Import the numpy library and use .arange to create a list of 50 numbers as our random states. Use each of these 50 numbers as the random state and take average of the respective accuracy score. This averages to a 78.8% accuracy.

Now we can go back to the dataframe and see what additional features we can use to improve this performance. In the baseline model we dropped the columns Name, Ticket, and Cabin, so maybe there is some useful information we can extract from these 3 columns and incorporate them back into the model.

We know there’s a lot of missing values in the Cabin column, so it’s probably not worth it to try to fill all of them like we did with the Age column. However, there might be reasons behind why a certain passenger is missing his/her Cabin entry, and this reason could be correlated to whether he/she survives. We can check if this is a possibility by printing out the survival ratio of people with missing Cabin entries and those without:

Survival rate comparisons between passengers with missing vs. non-missing Cabin entries

The survival rate is indeed very different, so it looks like this is a feature we want. To use this as a feature:

df["is missing cabin"] = df["Cabin"].isna()

Note that boolean features are also accepted as our model input.

Then run the loop again. This time, our average score comes out to be 78.7%, which suggests that there is no improvements to the model by adding this feature.

Let’s try adding something else.

In the Kaggle discussion section, there’s often people sharing code that were useful in improving their model. You can find inspiration from these discussion and add features to your own model. For example, I found a piece of code that handles the name column here:

Essentially, there is a title succeeding every passenger’s name in the name column. Some common titles are Mr., Mrs., Miss., and these titles are always followed by a “.” character. To extract title:

df['Title'] = df['Name'].str.extract(' ([A-Za-z]+)\.', expand=False)

And our Title column will look like: