The correct use of ColumnTransformer() in the Kaggle Titanic competition
Tags: Python, Kaggle, Data Science, Columntransformer, Sklearn

Sklearn has got to be one of my favourite libraries in Python. This library is not static, as it frequently introduces new functions in its updated releases. The current version of sklearn is 0.23.2 where it has introduced new features, such as: visual representation of estimators, improvements to K-means, improvements to gradient boosting, new generalised linear models, and sample weight support for existing regressors. It is always a good idea, therefore to find out the new features of this fascinating library and incorporate them into programming in an attempt to improve performance.

One new feature in sklearn’s version 0.20 is ColumnTransformer(). This function applies transformers to columns of an array or pandas DataFrame. It allows different columns or column subsets of the input to be transformed separately and the features generated by each transformer will be concatenated to form a single feature space.

I had already decided that I was going to try out ColumnTransformer() on the Kaggle Titanic competition datasets I have been working on to see if this tool will improve the accuracy of my predictions, but I found quite by accident that sklearn has already used ColumnTransformer on their own version Titanic dataset, with the link and user documentation here:- https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html#sphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py

The Kaggle Titanic datasets I use have been separated out into train and test datasets and I have employed some techniques different to those used by sklearn, so I nevertheless decided to see if I could improve accuracy on the competition question I have been working on for quite some time now. The Kaggle Titanic competition question can be found here:- https://www.kaggle.com/competitions

The problem statement for this competition reads:-

“The sinking of the Titanic is one of the most infamous shipwrecks in history.

On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.

While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.

In this challenge, we ask you to build a predictive model that answers the question: “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc).”

In order to solve this problem, the first thing that needs to be done is to import the libraries that are going to be used to write the program.

Pandas is a free software library written for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.

NumPy is a free python library used for working with arrays. It also has functions for working in domain of linear algebra, fourier transform, and matrices. NumPy was created in 2005 by Travis Oliphant.

Matplotlib is a free plotting library for the Python programming language and its numerical mathematics extension NumPy.

Seaborn is a free Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.

Once the main libraries have been imported, os needs to be imported so the Kaggle datasets can be retrieved:-

The train and test datasets are then loaded and read into the program.

Once the datasets are read, a check for any null values needs to be carried out. In the train dataset there were null values in “Age”, “Cabin” and “Embarked”. In the test dataset the null values are found in “Age”, “Fare” and “Cabin”:-

ID_test is created from test.Passenger.ID because this variable will be needed at the end of the program when the submission dataframe is created.

Three columns are deleted from the train and test datasets because they are not needed for the computations. The columns being deleted are “PassengerID”, “Cabin” and “Ticket”:-

A new column, “Title”, is created from the “Name” column. This column is created by defining a local function, lambda, and splitting the characters between “,” and “.”:-

The titles that had been derived from the name are now encoded to assign a numerical value to each title. These numerical values are then mapped into the column, “Title”:-

A new column, “Family” is created by adding “SibSp” to “Parch”:-

It is at this point that ColumnTransformer() is put into place.

Numeric features are “Age”, “Fare” and “Family”. These numeric_features are put into a pipeline called numeric_transformer, where the missing values are imputed and then scaled.

Categorical features are “Embarked”, “Sex”, “Pclass”, and “Title”. These categorical_features are put into a pipeline where the missing values are imputed and one hot encoded.

The column transformer is created by the numeric_transformer and categorial_transformer, being called a preprocessor:-

The X and y variables are then defined. The target value, being y, is train.Survived.

X variable is composed of the train dataset with “Survived”, “Name”, “SibSp”, and “Parch” dropped.

X_text is composed of the test dataset with “Name”, “SibSp” and “Parch” being dropped:-

The train dataset is then split up for training and validation. Because this is a classification problem, stratify is set to y:-

The variable class_weights is assigned because this competition question is a classification problem. Setting the model up for the correct class weight will help to improve accuracy:-

I then selected the model, in this case being the BaggingClassifier() with the base estimator being RandomForestClassifier(). Using this ensemble, I was able to achieve a 94.38% accuracy.

When I predicted on the validation set, I was able to achieve an accuracy of 80%! With such a high score, I hoped my accuracy when predicting on the test set would be higher, such as at least 78%:-

I predicted on the test dataset and built a dataframe that would be converted to a .csv file, which I could use to submit my predictions to Kaggle:-

Sadly, when I submitted my predictions to Kaggle, the accuracy of my model had not made any significant improvement, hovering at 77%.

While I had hoped using ColumnTransformer would improve accuracy when making predictions, this did not appear to be the case. I will continue to use ColumnTransformer, however, to keep my programming in line with sklearn standards.

The code for this program can be found in its entirety on the Kaggle competition page, the web address being found here:- https://www.kaggle.com/tracyporter/titanic-columntransformer?scriptVersionId=47800763