Building A Logistic Regression in Python, Step by Step
By: Susan Li
Tags: Machine Learning, Data Science, Python, Logistic Regression, Classification

Logistic Regression is a Machine Learning classification algorithm that is used to predict the probability of a categorical dependent variable. In logistic regression, the dependent variable is a binary variable that contains data coded as 1 (yes, success, etc.) or 0 (no, failure, etc.). In other words, the logistic regression model predicts P(Y=1) as a function of X.

Logistic Regression Assumptions

Binary logistic regression requires the dependent variable to be binary.

For a binary regression, the factor level 1 of the dependent variable should represent the desired outcome.

Only the meaningful variables should be included.

The independent variables should be independent of each other. That is, the model should have little or no multicollinearity.

The independent variables are linearly related to the log odds.

Logistic regression requires quite large sample sizes.

Keeping the above assumptions in mind, let’s look at our dataset.

Data

The dataset comes from the UCI Machine Learning repository, and it is related to direct marketing campaigns (phone calls) of a Portuguese banking institution. The classification goal is to predict whether the client will subscribe (1/0) to a term deposit (variable y). The dataset can be downloaded from here.

import pandas as pd

import numpy as np

from sklearn import preprocessing

import matplotlib.pyplot as plt

plt.rc("font", size=14)

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import train_test_split

import seaborn as sns

sns.set(style="white")

sns.set(style="whitegrid", color_codes=True)

The dataset provides the bank customers’ information. It includes 41,188 records and 21 fields.

Figure 1

Input variables

age (numeric) job : type of job (categorical: “admin”, “blue-collar”, “entrepreneur”, “housemaid”, “management”, “retired”, “self-employed”, “services”, “student”, “technician”, “unemployed”, “unknown”) marital : marital status (categorical: “divorced”, “married”, “single”, “unknown”) education (categorical: “basic.4y”, “basic.6y”, “basic.9y”, “high.school”, “illiterate”, “professional.course”, “university.degree”, “unknown”) default: has credit in default? (categorical: “no”, “yes”, “unknown”) housing: has housing loan? (categorical: “no”, “yes”, “unknown”) loan: has personal loan? (categorical: “no”, “yes”, “unknown”) contact: contact communication type (categorical: “cellular”, “telephone”) month: last contact month of year (categorical: “jan”, “feb”, “mar”, …, “nov”, “dec”) day_of_week: last contact day of the week (categorical: “mon”, “tue”, “wed”, “thu”, “fri”) duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y=’no’). The duration is not known before a call is performed, also, after the end of the call, y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact) pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted) previous: number of contacts performed before this campaign and for this client (numeric) poutcome: outcome of the previous marketing campaign (categorical: “failure”, “nonexistent”, “success”) emp.var.rate: employment variation rate — (numeric) cons.price.idx: consumer price index — (numeric) cons.conf.idx: consumer confidence index — (numeric) euribor3m: euribor 3 month rate — (numeric) nr.employed: number of employees — (numeric)

Predict variable (desired target):

y — has the client subscribed a term deposit? (binary: “1”, means “Yes”, “0” means “No”)

The education column of the dataset has many categories and we need to reduce the categories for a better modelling. The education column has the following categories:

Figure 2

Let us group “basic.4y”, “basic.9y” and “basic.6y” together and call them “basic”.

data['education']=np.where(data['education'] =='basic.9y', 'Basic', data['education'])

data['education']=np.where(data['education'] =='basic.6y', 'Basic', data['education'])

data['education']=np.where(data['education'] =='basic.4y', 'Basic', data['education'])

After grouping, this is the columns:

Figure 3

Data exploration

Figure 4

count_no_sub = len(data[data['y']==0])

count_sub = len(data[data['y']==1])

pct_of_no_sub = count_no_sub/(count_no_sub+count_sub)

print("percentage of no subscription is", pct_of_no_sub*100)

pct_of_sub = count_sub/(count_no_sub+count_sub)

print("percentage of subscription", pct_of_sub*100)

percentage of no subscription is 88.73458288821988

percentage of subscription 11.265417111780131

Our classes are imbalanced, and the ratio of no-subscription to subscription instances is 89:11. Before we go ahead to balance the classes, let’s do some more exploration.

Figure 5

Observations:

The average age of customers who bought the term deposit is higher than that of the customers who didn’t.

The pdays (days since the customer was last contacted) is understandably lower for the customers who bought it. The lower the pdays, the better the memory of the last call and hence the better chances of a sale.

Surprisingly, campaigns (number of contacts or calls made during the current campaign) are lower for customers who bought the term deposit.

We can calculate categorical means for other categorical variables such as education and marital status to get a more detailed sense of our data.