Logistic Regression (Complete Theory and Python Implementation)
By: Sameer Bairwa
Tags: Machine Learning, Python, Data Science, Statistics, Logistic Regression

In this post, you will discover the logistic regression algorithm for machine learning.

Contents…

• Overview • Real-World Examples • The Logistic Equation • Assumptions • Logistic Regression (Python Implementation)

data

input-variable

Predict Variable (desired target)

Data Exploration

Observations

Visualizations

Feature Selection

Implementing the Model

Logistic Regression Model Fitting

Predicting the test set results and calculating the accuracy

Cross-Validation

Confusion Matrix

ROC Curve

After reading this post you will know:

The many names and terms used when describing logistic regression (like log odds and logit).

The representation used for a logistic regression model.

Techniques used to learn the coefficients of a logistic regression model from data.

How to make predictions using a learned logistic regression model.

Where to go for more information if you want to dig a little deeper.

Logistic regression is a predictive modeling algorithm that is used when the Y variable is binary categorical. That is, it can take only two values like 1 or 0.

The goal is to determine a mathematical equation that can be used to predict the probability of event 1 happening.

Once the equation is established, it can be used to predict the Y when only the X’s are known.

In linear regression, the Y variable is always continuous If suppose, the Y variable was categorical, you cannot use the linear regression model for it.

So what would you do when Y is a categorical variable with 2 classes?

Logistic regression can be used to model and solve such problems, also called Binary Classification problems.

A key point to note here is that Y can have 2 classes only and not more than that.

Yet, Logistic regression is a classic predictive modeling technique and remains a popular choice for modeling binary categorical variables.

Another advantage of logistic regression is that it computes a prediction probability score of an event. More on that when you start building the models.

Real-World Examples

Some real-world examples of binary classification problems (Logistic Regression)

Spam Detection : Predicting if an email is Spam or not

: Predicting if an email is Spam or not Credit Card Fraud : Predicting if a given credit card transaction is fraud or not

: Predicting if a given credit card transaction is fraud or not Health : Predicting if a given mass of tissue is benign or malignant

: Predicting if a given mass of tissue is benign or malignant Marketing: Predicting if a given user will buy an insurance product or not

Predicting if a given user will buy an insurance product or not Banking: Predicting if a customer will default on a loan.

Why not Linear Regression?

When the response variable has only 2 possible values, it is desirable to have a model that predicts the value either as 0 or 1 or as a probability score that ranges between 0 and 1.

Linear regression does not have this capability.

Because, If you use linear regression to model a binary response variable, the resulting model may not restrict the predicted Y values within 0 and 1.

This is where logistic regression comes into play. In logistic regression, you get a probability score that reflects the probability of the occurrence of the event.

The Logistic Equation

Logistic regression achieves this by taking the log odds of the event

ln⁡(P/(1 -P))

where P is the probability of the event. So P always lies between 0 and 1.

Taking exponent on both sides of the equation gives:

Logistic Function

The logistic function, also called the sigmoid function was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It’s an S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits.

1 / (1 + e^-value)

Assumptions

Binary logistic regression requires the dependent variable to be binary.

For a binary regression, the factor level 1 of the dependent variable should represent the desired outcome.

Only meaningful variables should be included.

The independent variables should be independent of each other. That is, the model should have little or no multicollinearity.

The independent variables are linearly related to the log odds.

Logistic regression requires quite large sample sizes.

Logistic Regression (Python Implementation)

Data

The dataset comes from the UCI Machine Learning repository , and it is related to direct marketing campaigns (phone calls) of a Portuguese banking institution.

, and it is related to direct marketing campaigns (phone calls) of a Portuguese banking institution. The classification goal is to predict whether the client will subscribe (1/0) to a term deposit (variable y).

Dataset Link: https://drive.google.com/file/d/1o4RjXH5Vu1ZL3cJEk4D-wXWGxzbGexaB/view?usp=sharing

import pandas as pd

import numpy as np

from sklearn import preprocessing

import matplotlib.pyplot as plt

plt.rc("font", size=14)

from sklearn.linear_model import LogisticRegression

from sklearn.cross_validation import train_test_split

import seaborn as sns

sns.set(style="white")

sns.set(style="whitegrid", color_codes=True)

The dataset provides the bank customers’ information. It includes 41,188 records and 21 fields.

Input Variables:

age (numeric) job: type of job (categorical: “admin”, “blue-collar”, “entrepreneur”, “housemaid”, “management”, “retired”, “self-employed”, “services”, “student”, “technician”, “unemployed”, “unknown”) marital: marital status (categorical: “divorced”, “married”, “single”, “unknown”) education (categorical: “basic.4y”, “basic.6y”, “basic.9y”, “high.school”, “illiterate”, “professional.course”, “university.degree”, “unknown”) default: has credit in default? (categorical: “no”, “yes”, “unknown”) housing: has a housing loan? (categorical: “no”, “yes”, “unknown”) loan: has a personal loan? (categorical: “no”, “yes”, “unknown”) contact: contact communication type (categorical: “cellular”, “telephone”) month: last contact month of the year (categorical: “jan”, “feb”, “mar”, …, “nov”, “dec”) day_of_week: last contact day of the week (categorical: “mon”, “tue”, “wed”, “thu”, “fri”) duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y=’no’). The duration is not known before a call is performed, also, after the end of the call, y is known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model campaign: number of contacts performed during this campaign and for this client (numeric, includes the last contact) pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means the client was not previously contacted) previous: number of contacts performed before this campaign and for this client (numeric) poutcome: outcome of the previous marketing campaign (categorical: “failure”, “nonexistent”, “success”) emp.var.rate: employment variation rate — (numeric) cons.price.idx: consumer price index — (numeric) cons.conf.idx: consumer confidence index — (numeric) euribor3m: euribor 3 month rate — (numeric) nr.employed: number of employees — (numeric)

Predict Variable (desired target)

y — has the client subscribed a term deposit? (binary: “1”, means “Yes”, “0” means “No”)

The education column of the dataset has many categories and we need to reduce the categories for better modeling.

The education column has the following categories:

Let us group “basic.4y”, “basic.9y” and “basic.6y” together and call them “basic”.

data['education']=np.where(data['education'] =='basic.9y', 'Basic', data['education'])

data['education']=np.where(data['education'] =='basic.6y', 'Basic', data['education'])

data['education']=np.where(data['education'] =='basic.4y', 'Basic', data['education'])

After grouping, this is the columns:

Data Exploration

There are 36548 no’s and 4640 yes’s in the outcome variables.

Let’s get a sense of the numbers across the two classes.

Observations

The average age of customers who bought the term deposit is higher than that of the customers who didn’t.

The pdays (days since the customer was last contacted) are understandably lower for the customers who bought it. The lower the pdays, the better the memory of the last call and hence the better chances of a sale.

Surprisingly, campaigns (number of contacts or calls made during the current campaign) are lower for customers who bought the term deposit.

We can calculate categorical means for other categorical variables such as education and marital status to get a more detailed sense of our data.

Visualizations

The frequency of purchase of the deposit depends a great deal on the job title. Thus, the job title can be a good predictor of the outcome variable.

%matplotlib inline

pd.crosstab(data.job,data.y).plot(kind='bar')

plt.title('Purchase Frequency for Job Title')

plt.xlabel('Job')

plt.ylabel('Frequency of Purchase')

plt.savefig('purchase_fre_job')

The marital status does not seem a strong predictor for the outcome variable.

table=pd.crosstab(data.marital,data.y)

table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)

plt.title('Stacked Bar Chart of Marital Status vs Purchase')

plt.xlabel('Marital Status')

plt.ylabel('Proportion of Customers')

plt.savefig('mariral_vs_pur_stack')

Education seems a good predictor of the outcome variable.

table=pd.crosstab(data.education,data.y)

table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)

plt.title('Stacked Bar Chart of Education vs Purchase')

plt.xlabel('Education')

plt.ylabel('Proportion of Customers')

plt.savefig('edu_vs_pur_stack')

The day of the week may not be a good predictor of the outcome.

pd.crosstab(data.day_of_week,data.y).plot(kind='bar')

plt.title('Purchase Frequency for Day of Week')

plt.xlabel('Day of Week')

plt.ylabel('Frequency of Purchase')

plt.savefig('pur_dayofweek_bar')

The month might be a good predictor of the outcome variable.

pd.crosstab(data.month,data.y).plot(kind='bar')

plt.title('Purchase Frequency for Month')

plt.xlabel('Month')

plt.ylabel('Frequency of Purchase')

plt.savefig('pur_fre_month_bar')

Most of the customers of the bank in this dataset are in the age range of 30–40.

data.age.hist()

plt.title('Histogram of Age')

plt.xlabel('Age')

plt.ylabel('Frequency')

plt.savefig('hist_age')

Poutcome seems to be a good predictor of the outcome variable.

pd.crosstab(data.poutcome,data.y).plot(kind='bar')

plt.title('Purchase Frequency for Poutcome')

plt.xlabel('Poutcome')

plt.ylabel('Frequency of Purchase')

plt.savefig('pur_fre_pout_bar')

Create dummy variables

That is variables with only two values, zero and one.

cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']

for var in cat_vars:

cat_list='var'+'_'+var

cat_list = pd.get_dummies(data[var], prefix=var)

data1=data.join(cat_list)

data=data1 cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']

data_vars=data.columns.values.tolist()

to_keep=[i for i in data_vars if i not in cat_vars]

Our final data columns will be:

data_final=data[to_keep]

data_final.columns.values

data_final_vars=data_final.columns.values.tolist()

y=['y']

X=[i for i in data_final_vars if i not in y]

Feature Selection

Recursive Feature Elimination (RFE) is based on the idea to repeatedly construct a model and choose either the best or worst performing feature, setting the feature aside and then repeating the process with the rest of the features.

This process is applied until all features in the dataset are exhausted.

The goal of RFE is to select features by recursively considering smaller and smaller sets of features.

from sklearn import datasets

from sklearn.feature_selection import RFE

from sklearn.linear_model import LogisticRegressionlogreg = LogisticRegression()rfe = RFE(logreg, 18)

rfe = rfe.fit(data_final[X], data_final[y] )

print(rfe.support_)

print(rfe.ranking_)

The RFE has helped us select the following features:

“previous”, “euribor3m”, “job_blue-collar”, “job_retired”, “job_services”, “job_student”, “default_no”, “contact_telephone”, “month_apr”, “month_aug”, “month_mar”, “month_may”, “month_nov”, “day_of_week_mon”, “day_of_week_wed”, “poutcome_failure”, “poutcome_nonexistent”, “poutcome_success”.

pref_indexes = list(np.where(rfe.ranking_ ==1)[0]) cols = list(np.asarray(X)[pref_indexes])

X=data_final[cols]

y=data_final['y']

Implementing the Model

import statsmodels.api as sm

logit_model=sm.Logit(y,X)

result=logit_model.fit()

print(result.summary())

The p-values for most of the variables are smaller than 0.05, therefore, most of them are significant to the model.

Logistic Regression Model Fitting

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

from sklearn.linear_model import LogisticRegression

from sklearn import metrics

logreg = LogisticRegression()

logreg.fit(X_train, y_train)

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class=’ovr’, n_jobs=1, penalty=’l2', random_state=None, solver=’liblinear’, tol=0.0001,

verbose=0, warm_start=False

Predicting the test set results and calculating the accuracy

y_pred = logreg.predict(X_test)

print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))

Accuracy of logistic regression classifier on test set: 0.90

Cross-Validation

Cross-validation attempts to avoid overfitting while still producing a prediction for each observation dataset.

We are using 10-fold Cross-Validation to train our Logistic Regression model.

from sklearn import model_selection

from sklearn.model_selection import cross_val_score

kfold = model_selection.KFold(n_splits=10, random_state=7)

modelCV = LogisticRegression()

scoring = 'accuracy'

results = model_selection.cross_val_score(modelCV, X_train, y_train, cv=kfold, scoring=scoring)

print("10-fold cross validation average accuracy: %.3f" % (results.mean()))

10-fold cross-validation average accuracy: 0.898

The average accuracy remains very close to the Logistic Regression model accuracy; hence, we can conclude that our model generalizes well.

Confusion Matrix

from sklearn.metrics import confusion_matrix

confusion_matrix = confusion_matrix(y_test, y_pred)

print(confusion_matrix)

[[10873 108]

[ 1104 272]]

The result is telling us that we have 10873+272 (=11,145) correct predictions and 1104+108 (=1,212) incorrect predictions.

ROC Curve

from sklearn.metrics import roc_auc_score

from sklearn.metrics import roc_curve

logit_roc_auc = roc_auc_score(y_test, logreg.predict(X_test))

fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(X_test)[:,1])

plt.figure()

plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)

plt.plot([0, 1], [0, 1],'r--')

plt.xlim([0.0, 1.0])

plt.ylim([0.0, 1.05])

plt.xlabel('False Positive Rate')

plt.ylabel('True Positive Rate')

plt.title('Receiver operating characteristic')

plt.legend(loc="lower right")

plt.savefig('Log_ROC')

plt.show()

The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers. The dotted line represents the ROC curve of a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner).

References

[1]http://www.holehouse.org/mlclass/06_Logistic_Regression.html

[2]http://machinelearningmastery.com/logistic-regression-tutorial-for-machine-learning

[3]https://scilab.io/machine-learning-logistic-regression-tutorial/

[4]https://github.com/perborgen/LogisticRegression/blob/master/logistic.py

[5]http://neuralnetworksanddeeplearning.com/chap3.html

[6]http://math.stackexchange.com/questions/78575/derivative-of-sigmoid-function-sigma-x-frac11e-x

[7]https://en.wikipedia.org/wiki/Monotoniconotonic_function

[8]http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression>

[9]https://en.wikipedia.org/wiki/Softmax_function

That’s all about Logistic regression Theory and Python Implementation.