A comparison of sklearn and statsmodel’s logistic regression function
Tags: Statsmodels, Logistic Regression, Python, Data Science, Sklearn

I have to say that I had not generally spent a lot of time in the Python library, statsmodels, because I felt it was a bit scary. Statsmodels is written more in the theme of R, while sklean, a similar library seems to me to be more user friendly. I have conducted a bit of research on the subject and have found that Scikit-learn offers a lot of simple, easy to learn algorithms that pretty much only require the input data to be organized in the right way before classification, regression, or clustering can be performed. The pipelines provided in sklearn even make the process of transforming data easier. Statsmodels, on the other hand, offers superior statistics and econometric tools, so when a variety of linear regression models, mixed linear models, or regression with discrete dependent variables are needed, statsModels has options.

With the basics of these two Python libraries covered, I felt it would be a good idea to compare the difference between sklearn and statsmodels by running functions from both libraries on a dataset. During my research, I have found that other individuals have endeavoured to compare sklearn to statsmodels, but I decided that for the sake of science, I will contribute my comparison too.

I have chosen a very common dataset to use in this experiment, being the Titanic dataset from Kaggle, the link being found here:- Titanic — Machine Learning from Disaster | Kaggle

The problem statement for the Kaggle competition question reads as follows:-

“The sinking of the Titanic is one of the most infamous shipwrecks in history.

On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.

While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.

In this challenge, we ask you to build a predictive model that answers the question: “what sorts of people were more likely to survive?” using passenger data (ie name, age, gender, socio-economic class, etc).”

When I am working on the Titanic dataset I generally write the program in Google Colab and then transfer the code to the Jupyter Notebook in Kaggle, but on this occasion I chose not to transfer the program to Kaggle because I have achieved higher accuracies using other models in the past. It is for this reason that I decided not to check the accuracy of the test dataset on Kaggle, but at a later time I may decide to do so.

Without further ado, I will begin the comparison:-

After creating the program in Google Colab, I imported the libraries that are already pre-installed on the online Jupyter Notebook. I initially installed pandas, numpy, matplotlib, and seaborn.

I then loaded and read the train and test dataset, although I will only be predicting on the train dataset in this comparison:-

I checked for any null values and found there are null values in three columns in both the train and test datasets. Two of the columns are numeric and two are categorical in nature:-

I decided to impute the numeric columns by replacing null values with the median value of the associated columns. I then imputed the categorical values by replacing null values with the most commonly used value, being mode()[0].

Because the models want to see numeric data, I ordinal encoded the categorical columns, which is merely converting object data to numeric data for ease of computation. This preprocessing step is generally standard practice in machine learning because most models will not train and fit on categorical data:-

I then defined the X, y and X_test variables. The label, being y, is comprised of the “Survived” column in the train dataset. The independent variable, being X, is comprised of the train dataset with the following columns dropped: “PassengerId”, “Survived”, “Name”, “Cabin”, and “Ticket”. X_test is designed to have the same format as X, except “Survived” is not dropped because there was no such column in the test dataset:-

I then used sklearn’s StandardScaler() to scale the data in the X and X_test datasets to an appropriate level for use in the model.

I put the valuation target on a counter and found that 55 examples were 0, meaning people perished, and 35 examples were 1, meaning people survived.

I then defined the variable, class_weights, which will be used by the model because there is a slight class imbalance in the train dataset:-

At this point I selected sklearn’s model, LogisticRegression() as the first comparator. When I set the model up with the correct parameters, I achieved an accuracy of 79.40%.

When I predicted on the validation set, I achieved an accuracy of 72.22%, which is a lower accuracy than I have achieved using other models, such as, say, Ransom Forest.

I created a confusion matrix and counted the examples, finding 60 examples were 0 and 30 examples were 1.

The second model I chose was statsmodels’ Logit(), which is the equivalent of sklearn’s LogisticRegression(). Statsmodels is coded differently than sklearn, so I was unable to record an accuracy then training and fitting the training set.

I predicted on the validation set and achieved an accuracy of 75.56%, which is 3 percentage points higher than the accuracy I achieved using sklearn’s LogisticRegression().

I created a confusion matrix and put the prediction on a counter, finding that 57 examples were 0 and 33 examples were 1.

In summary, statsmodel’s Logit() performed better than sklkearn’s LogisticRegression() so it is best to train, fit and predict on both models and then select the one that affords the highest level of accuracy.

When I researched the reason why statsmodels’ Logit() performs better than sklearn’s LogisticRegression() I found the reason for this is because sklearn’s parameter’s are tighter than statsmodels. There are ways of getting around this by tuning the parameters, i.e. LogisticRegression(C=100, penalty=’none’). I tried tuning the parameters to make them less tight, but this did not affect the model’s accuracy.

The code for this post can be found in its entirety in my personal GitHub account, the link being found here:- Titanic-Datasets/Kaggle_Titanic_Log_Reg_SKL_vs_SM.ipynb at master · TracyRenee61/Titanic-Datasets (github.com)