What happened when I used Stacking() on the Kaggle Titanic competition
Tags: Python, Kaggle, Data Science, Titanic Dataset, Machine Learning

Stacking is a type of ensemble machine learning algorithm. This function in sklearn library combines the best predictors from two or more functions in library. The benefit of using stacking is that this algorithm can harness the range of well performing models and make predictions that have the potential for obtaining a better outcome than if these models were used singly.

The stacking algorithm is designed to improve modelling performance, although it is not necessarily guaranteed. Achieving an improvement in accuracy depends on the complexity of the problem and whether it is sufficiently well represented by the training data and complex enough so there is more to learn by by combining operations, or stacking.

With the potential improvement in accuracy in mind, I chose a real life training set about the survivors of the Titanic that sank in the early 20th century. The datasets for this example can be found on the Kaggle competition page:- https://www.kaggle.com/c/titanic/data

Excerpts of the Problem Statement of the Kaggle competition read as follows:-

“On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.

While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.

In this challenge, we ask you to build a predictive model that answers the question: “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc).”

I would like to say that I have worked on the Titanic datasets for quite some time because they are small and training the models will not crash my computer, which is what quite often happens when I am working with larger datasets. The idea behind the Titanic and other competitions is that the budding data scientist is supposed to move up the leaderboard on this competition, but all of the different algorithms I have experimented with have only led to incremental improvements. With this knowledge in mind, let’s see what happens when stacking is employed.

The first thing I did was to import the libraries that are needed to solve this computational problem. Pandas and numpy are the mainstays of Python, so they were the first two libraries to be imported.

Because this program was written in Kaggle, it was also necessary to import the operating system used by Kaggle and print out the data files in its directory:-

I then read the train and test data files that would be used for this calculation:-

I checked for any null values and found they were present in the following columns: “Age”, “Cabin”, “Embarked” and “Fare”. All of these columns would need to be imputed before I could train and then fit the datasets into the model:-

I decided to create variables, “ID_train” and “ID_test” , which stored the data in the “PassengerId”. I decided to drop the “PassengerId”, “Cabin”, and “Ticket” columns from the train and test sets because they were no longer needed:-

I used KNNImputer() to impute the missing values in the columns that had numeric data, being “Age” and “Fare”:-

I then replaced the missing values of the column that had objects in it, “Embarked”, with the most commonly used value, stated as mode():-

Once the values had been imputed, I encoded the columns that contained object data, using OrdinalEncoder():-

As part of the data preprocessing, I created a new column entitled “Title” and extracted the title from the name. I accomplished this by creating a user defined function, lamba, and split the string at “,” and “.” to extract the title:-

I then encoded the titles, using mapping, to assign each title a number, which is what the computer wants to see when it fits the data to the various functions in the algorithm:-

I then created bins to assign a range of numbers to a category, being ‘child’, ‘young adult’, ‘middle aged’, and ‘pensioner’. This range was put into another column I created, “Age_Range”:-

I then mapped the age range, assigning numeric values to each category:-

I created a new column. “Family”, by combining “SibSp” and “Parch”:-

I created a new column, “Fare_Range” and put the fares into categories:-

I then assigned a number to each category in “Fare_Range” and mapped this to make the column numeric:-

After preprocessing the data, I set up the X, y and X_test values. I created a variable, features, and put all of the columns I was intending to use as the X and X_test variables in it. I then used the “Survived” column in the train dataset and used it as the y variable.

Once the X, y, and X_test variables had been established, I used train_test_split() to split the train dataset up for training and validation. Since this is a classification problem, I set stratify=y:-

I then used StandardScaler() to scale the data in the training, validation and testing sets on the same scale as the target variable:-

Because the Titanic dataset has a mild class imbalance, I created a variable, class_werights, that will be used to balance the classes during the modelling process:-

The data is now ready to be trained, so I used the Stacking() function to create an ensemble of models to be stacked. Stacked generalization is a method for combining estimators to reduce their biases. More precisely, the predictions of each individual estimator are stacked together and used as input to a final estimator to compute the prediction. In this instance, the final estimator od LogisticRegression().

When I trained and fitted this model, I achieved an accuracy of 85.02%:-

Once the training set had been trained and fitted, I predicted on the validation set and obtained an accuracy of 77.78%:-

I then predicted on the test dataset and put the predictions in a dataframe in accordance with Kaggle’s competition specifications:-

When I submitted the predictions to Kaggle, i received a public score of 77.511%, being only a minor improvement from my previous score. Obviously much intensive work needs to be performed to increase the accuracy of this dataset:-

The code for this dataset can be found in my Jupyter notebook on the Kaggle Titanic competition page, being:- https://www.kaggle.com/tracyporter/titanic-stacking?scriptVersionId=47458851