What happened when I predicted on the Titanic dataset with a made from scratch Logistic Regression function
Tags: Logistic Regression, Data Science, Artificial Intelligence, Python, Machine Learning

What happened when I predicted on the Titanic dataset with a made from scratch Logistic Regression function Tracyrenee Follow Dec 19, 2020 · 7 min read

Because there are so many libraries in Python, the programming language I am studying, it is easy to lose sight of the fact that these libraries are comprised of functions that somebody has written. Sklearn, the library I research a lot, is completely full of supervised and unsupervised learning functions that someone has written and would like to be given credit for. In fact, the sklearn website states that if anyone uses their work to please cite this, which I endeavour to do.

Not only are functions made by individuals, they are also put together by teams and even businesses. CatBoost, a competitor of the award winning XGBoost, was actually made by Yandex Technologies, who made the algorithms associated with this library open source so everybody can use them.

Whilst studying data science I have entered several competitions and experimented with many models. Through my own empirical investigations, I have come to the conclusion that winners of data science competitions need to be able to perform programming above and beyond merely using a function from one of the numerous python libraries because most people with enough training can do that.

It is therefore important that anyone serious about advancing in the field of data science learn how to program their own functions. I was therefore inspired to have a stab at using a made from scratch logistic regression function and found several articles where individuals have displayed their own made from scratch logistic regression functions. I was inspired to use a logistic regression function on one website, but I would like to say that the person who wrote the function used simulated data instead of real datasets, which made his predictions much easier to achieve. In addition, he did not split the dataset up for training and testing, which invalidated the accuracy recorded. Despite this, I nevertheless decided to modify and use this made from scratch logistic regression function, the link being found here:- Logistic Regression from Scratch in Python — nick becker (beckernick.github.io)

I prefer to use existing datasets for my experimentation, so decided to use Kaggle’s Titanic dataset for this purpose. I find the Titanic dataset a good dataset to carry out experimentation on because I have been unable to achieve any accuracy above 78% because of the way the data is presented in the dataset. I keep trying, though, and maybe one day my persistence will pay off. Kaggle’s Titanic dataset can be found at this link:- Titanic — Machine Learning from Disaster | Kaggle

The problem statement of the Titanic dataset reads as follows:-

“The sinking of the Titanic is one of the most infamous shipwrecks in history.

On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.

While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.

In this challenge, we ask you to build a predictive model that answers the question: “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc).”

The first thing I did was to create an .ipynb file in Google Colab, a free online Jupyter Notebook that has several of the more common libraries already installed on it. As a result, I only needed to import the pre-installed libraries of pandas, numpy, matplotlib.pyplot and seaborn.

I then loaded and read the train and test datasets, which I had downloaded from the Kaggle website and then saved to my own personal GitHub account. In this post I am only going to focus on the train dataset because I made the decision not to submit the predictions from the test dataset to Kaggle. At some point I may decide to submit the predictions to Kaggle, so never say never:-

I checked for any null values and found three columns in the train dataset have missing values, being “Age”, “Cabin” and “Embarked”.

I imputed the missing cells in the categorical columns by replacing them with the most frequently used value, or “mode”. I replaced the missing values in the “Age” column with the median value:-

I used seaborn’s displot() function to plot the target column, being train.Survived:-

Once the missing values had been imputed, I used sklearn’s OrdinalEncoder() to convert the columns with categorical data to numeric data, which is what most models want to see when they are performing computations. The only model I am aware of that will make predictions on categorical data is CatBoost and I will not be using it in this post:-

Once all of the columns were comprised of numeric data, I defined the X and y variables, The dependant variable is train.Survived and the label “y” has been assigned this value. The independent values in train are assigned to the variable of X once train.Survived and train.ID have been dropped:-

Once X and y had been defined, I used sklearn’s train_test_split() function to split the X dataset up for training and validation.

I then used sklearn’s StandardScaler() to scale the data down to an acceptable level for use in the model when it trains, fits and predicts.

I put the validation set on a counter and found that 55 people perished and 35 people survived.

I then defined the variable, “class_weights”, which can be used by the model to balance the classes in the label:-

Once all of the preprocessing had been accomplished, I set about using the logistic regression function that had been made from scratch.

I started by creating a link function, which provides the relationship between the linear predictor and the mean of the distribution function, sigmoid().

I then created the function, log_likelihood(), which is viewed as a sum over all the training data. This function also contains a line of code that calculates the gradient of the log_likelihood.

The logistic_regression() function is built and it will integrate the work of the two previous functions to be able to formulate predictions on input data.

The weights were set, using the logistic_regression() function to train the model.

I predicted on the training data only to obtain an accuracy, which is 80.27%.

I then predicted on the validation dataset and attained an accuracy of 75.56%. When I put the prediction on a counter I found that the model predicted 63 people would perish and 27 people would survive:-

I then decided to use sklearn’s LogisticRegression() and achieved an accuracy of 79.40% when I trained and fitted the data.

When I predicted on the validation dataset I achieved an accuracy of 72.22%, being 3% less than the made from scratch logistic_regression() function. The reason for this is because sklearn has very tight parameters, which ultimately affects its accuracy. When I put the prediction on the counter, it revealed that 60 people were believed to have perished while 30 people were believed to have survived:-

I also decided to compare the made from scratch logistic_regression() to statsmodels’ Logot(). When I predicted on the validation set, an accuracy of 75.56% was achieved, which means logistic_regression() and Logit() both record the same rate of accuracy.

When I placed the prediction on a counter, the model predicted that 57 perished and 33 people survived:-

To summarise, all three of the above models need to be used when endeavouring to find the best accuracy. I do feel that any person who wishes to excel at data science and win competitions needs to improve his programming skills. Performing code reviews on other programmers’ codes is always a good start in honing one’s skills to the point where he can win a competition.

I have seen a few programs where logistic regression was used to score a very high accuracy, so perhaps made from scratch functions are a good starting point because they can be more easily modified than an off the shelf function provided by one of Python’s libraries, such as sklearn or statsmodels.

The code for this post can be found in its entirety in my personal GitHub account, the link being found here:- Titanic-Datasets/Kaggle_Titanic_Log_Reg_Homemade_vs_SKL_vs_SM.ipynb at master · TracyRenee61/Titanic-Datasets (github.com)