A Practical Guide To Logistic Regression in Python for Beginners
By: Akbar Husnoo
Tags: Logistic Regression, Artificial Intelligence, Python, Machine Learning

Logistic Regression’s roots date back to the 19th century when Belgian Mathematician, Pierre François Verhulst proposed the Logistic Function/Logistic Growth in a series of three papers for modelling population growth. It was later applied for modelling autocatalysis in chemistry by Friedrich Wilhelm Ostwald in 1883. Around 200 years later, Logistic Regression is now one of the most widely utilised statistical models in various fields including machine learning, economics, medical, etc.

1.0. Geez! Some Mathematical Definitions 😞

In brief, a logistic regression model uses the logistic function:

to squeeze the output of a linear equation between 0 to 1. The logistic curve is a common Sigmoid curve (S-shaped) as follows:

2.0. Important Assumptions 🧐

There are 4 major assumptions to consider before using Logistic Regression for modelling. These are:

The dependent/response/target variable MUST be binary or dichotomous: A data point must fit only one of the two categories. For instance, to predict whether a person has tumour — Yes(1), No (0). LACK of Multi-collinearity: Independent/predictor variables must have little to no collinearity amongst them meaning that they should be independent of each other. Variance Inflation Factor (VIF) is one of the simple tests that can be used to check for multi-collinearity. If the VIF score for a factor is above 5, it is better to remove one of the correlated independent variables to reduce redundancy. LARGE sample size: As with any statistical model, past data is the key for a robust model. Similarly, the larger the sample size, the better and more reliable is the result of the Logistic Regression Analysis. Log-odds Relationship: Independent variables must be linearly related to log-odds.

3.0. Step-by-step Python Code Guide 🤓

This section serves as a complete guide/tutorial for the implementation of logistic regression the Bank Marketing dataset. The goal of this machine learning task is to predict whether or not a client will subscribe to a term deposit. The dataset and code used for this article can be found this GitHub repository.

I. Import Libraries

import pandas as pd

import seaborn as sns

import matplotlib.pyplot as plt

import statsmodels

import sklearn

II. Understand Dataset



url_link = ' #access dataset from GitHub linkurl_link = ' https://raw.githubusercontent.com/akbarhusnoo/Logistic-Regression-Portuguese-Bank-Marketing/main/Portuguese%20Bank%20Marketing%20Dataset.csv' #read file

df_bank = pd.read_csv(url_link) #display top 5 rows

display(df_bank.head()) #display number of rows and columns

print('

Shape of dataset = ', df_bank.shape) #display list of attributes present in dataset

print('

List of Attributes:

', list(df_bank.columns))

Output 3(II) — Image by Author

This dataset consists of 21 attributes/columns and 41188 records/rows. The variables age, job, etc. are the list of features while the variable y — has the client subscribed a term deposit? is the target variable whereby 1 represents ‘yes’ and 0 represents ‘no’.

III. Can the target variable be used for logistic regression?

YES. This is because the target variable is binary/dichotomous.

IV. Check for Missing Values

There are several ways to handle the nuisance caused by missing values in a dataset. In this case, since the sample size is big enough, if missing values are present, those values can be discarded.

#check if there are missing values in dataset

print(df_bank.isnull().values.any())

Output 3(IV) — Image by Author

In our case, the dataset does not contain any missing values.

V. Data Exploration

#visualise the target variable

sns.countplot(x ='y', data = df_bank)

plt.show()

Output 3(V) — Image by Author

The classes 0 and 1 are highly imbalanced. Therefore, accuracy is not a good performance evaluation metric for this scenario.

VI. Encoding Categorical Variables

The categorical variables present in the dataset are encoded using label encoder.

#label encoding for all categorical variables in dataset

from sklearn.preprocessing import LabelEncoder label_encoder = LabelEncoder()

df_bank['job'] = label_encoder.fit_transform(df_bank['job'])

df_bank['marital'] = label_encoder.fit_transform(df_bank['marital'])

df_bank['education'] = label_encoder.fit_transform(df_bank['education'])

df_bank['default'] = label_encoder.fit_transform(df_bank['default'])

df_bank['housing'] = label_encoder.fit_transform(df_bank['housing'])

df_bank['loan'] = label_encoder.fit_transform(df_bank['loan'])

df_bank['contact'] = label_encoder.fit_transform(df_bank['contact'])

df_bank['month'] = label_encoder.fit_transform(df_bank['month'])

df_bank['day_of_week'] = label_encoder.fit_transform(df_bank['day_of_week'])

df_bank['poutcome'] = label_encoder.fit_transform(df_bank['poutcome'])

VII. Feature Engineering

A. Segment the feature set from the whole dataset

#segment dataset into features set

X = df_bank.loc[:, list(df_bank.columns)[0:20]]

B. Check for Multi-Collinearity

To check for multi-collinearity in the independent variables, the Variance Inflation Factor (VIF) technique is used. The variables with VIF score of >10 means that they are very strongly correlated. Therefore, they are discarded and excluded in the logistic regression model.

#calculate Variance Inflation Factor

from statsmodels.stats.outliers_influence import variance_inflation_factor

vif_scores = pd.DataFrame()

vif_scores["Attribute"] = X.columns



# calculating VIF for each feature

vif_scores["VIF Scores"] = [variance_inflation_factor(X.values, i) for i in range(len(X.columns))]



display(vif_scores)

Output 3(VII.B) — Image by Author

VIII. Split Data into Training and Testing

It is recommended to split the data into a 70–30 split whereby training dataset consists of 70% and testing dataset consists of 30%.

#segment dataset into significant features and target

X = df_bank.loc[:, ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'month', 'day_of_week', 'duration', 'campaign', 'previous']]

y = df_bank['y'] #split dataset into training and testing features and targets

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.7, test_size = 0.3)

IX. Fit Logistic Regression Model to Data

The train data is fed to the logistic regression model for training the latter.

#fit logistic regression model to data

from sklearn.linear_model import LogisticRegression logistic_regression_model = LogisticRegression()

logistic_regression_model.fit(X_train, y_train)

Output 3(IX) — Image by Author

X. Evaluating Logistic Regression Model

The test features are then fed to the logistic regression model. The performance metrices and confusion matrix are then computed as follows:

#evaluate model using test data

y_predicted = logistic_regression_model.predict(X_test)

display(y_predicted) #compute confusion matrix

from sklearn.metrics import confusion_matrix

confusion_matrix = confusion_matrix(y_test, y_predicted)

print('

The Confusion Matrix is as follows:

', confusion_matrix) #compute performance metrices

from sklearn.metrics import precision_score

from sklearn.metrics import roc_auc_score

print('

Area Under the Receiver Operating Characteristic Curve:', roc_auc_score(y_test, y_predicted))

print('

Precision:', precision_score(y_test, y_predicted))

Output 3(IX) — Image by Author

Therefore, the model predicts 11111 (10767+344) data correctly and 1246 (194+1052) data incorrectly.

4.0. References

[1]Statistics Solutions. (2013). What is Logistic Regression? — Statistics Solutions. [online] Available at: https://www.statisticssolutions.com/what-is-logistic-regression/.

[2]Jason Brownlee (2016). Logistic Regression for Machine Learning. [online] Machine Learning Mastery. Available at: https://machinelearningmastery.com/logistic-regression-for-machine-learning/.

[3]Rogel-Salazar, J. (2017). Data science and analytics with Python. Boca Raton, Fl: Crc Press, Taylor & Francis Group.

‌[4]Miller, T.W. (2015). Marketing data science : modeling techniques in predictive analytics with R and Python. Upper Saddle River: Financial Times/Prentice Hall.