Multivariate Logistic Regression in Python
By: Sowmya Krishnan
Tags: Logistic Regression, Machine Learning, Python, Classification Models, Confusion Matrix

Multivariate Logistic Regression in Python

A machine learning technique for classification

You probably use machine learning dozens of times a day without even knowing it. A simple web search on Google works so well because the ML software behind it has learnt to figure out which pages to be ranked and how. Similarly, you are saved from wading through a ton of spam in your email because your computer has learnt to distinguish between spam & a non-spam email.

Machine learning is a smart alternative to analyzing vast amounts of data. Based on the tasks performed and the nature of the output, you can classify machine learning models into three types:

Regression: where the output variable to be predicted is a continuous variable Classification: where the output variable to be predicted is a categorical variable Clustering: where there is no pre-defined notion of a label allocated to the groups/clusters formed.

A large number of important problem areas within the realm of classification — an important area of supervised machine learning. And despite the term ‘Regression’ in Logistic Regression — it is, in fact, one of the most basic classification algorithms. Unlike linear regression which outputs continuous number values, logistic regression uses the logistic sigmoid function to transform its output to return a probability value which can then be mapped to two or more discrete classes.

Types of Logistic Regression:

Binary (true/false, yes/no) Multi-class (sheep, cats, dogs) Ordinal (Job satisfaction level — dissatisfied, satisfied, highly satisfied)

Before we begin building a multivariate logistic regression model, there are certain conceptual pre-requisites that we need to familiarize ourselves with.

Sigmoid Function

Earlier we spoke about mapping values to probabilities. This can be achieved by calling the sigmoid function, which will map any real value into another value between 0 and 1. Machine learning uses this function to map predictions to probabilities.

Mathematically,

Where, f(x) = output between 0 and 1 (probability estimate)

Graphically,

Code:

def sigmoid(z):

return 1.0 / (1 + np.exp(-z))

Decision Boundary

To get a better sense of what a logistic regression hypothesis function computes, we need to know of a concept called ‘decision boundary’. The prediction function that we are using will return a probability score between 0 and 1. To map this score to a discrete class (positive/negative, true/false), we select a threshold value, say 0.5, above which we classify values into class 1 and below which the values will fall into class 2.

For instance, say the prediction function returns a value of 0.8, this would get classified as true/positive (as it is above the selected value of threshold). A value of 0.3, on the other hand, would get classified as false/negative. When dealing with multivariate logistic regression, we select the class with the highest predicted probability.

Odds and log odds

Logistic regression work with odds rather than proportions. The odds are simply calculated as a ratio of proportions of two possible outcomes. Let p be the proportion of one outcome, then 1-p will be the proportion of the second outcome.

Mathematically,

Odds = p/1-p

The statistical model for logistic regression is

log (p/1-p) = β0 + β1x

Multivariate Logistic Regression

To understand the working of multivariate logistic regression, we’ll consider a problem statement from an online education platform where we’ll look at factors that help us select the most promising leads, i.e. the leads that are most likely to convert into paying customers.

Note: Please follow the below given link (GitHub Repo) to find the dataset, data dictionary and a detailed solution to this problem statement.

Data Cleaning & Preparation

Once you load the necessary libraries and the dataset, let’s have a look at the first few entries using the head() command.

The shape commands tells us the dataset has a total of 9240 data points and 37 columns. Below listed are the name of the columns present in this dataset:

As you can see, most of the feature variables listed are quite intuitive. Please refer to the data dictionary to understand them better.

Further analysis reveals the presence of categorical variables in the dataset for which we would need to create dummy variables. Before that, we treat the dataset to remove null value columns and rows and variables that we think won’t be necessary for this analysis (eg, city, country) A quick check for the percentage of retained rows tells us that 69% of the rows have been retained which seems good enough.

Test-Train Split

The target variable for this dataset is ‘Converted’ which tells us if a past lead was converted or not, wherein 1 means it was converted and 0 means it wasn’t converted.

Import the test_train_split library and make a 70% train and 30% test split on the dataset.

Few numeric variables in the dataset have different scales, so scale these variables using the MinMax scaler. The variables will be scaled in such a way that all the values will lie between zero and one using the maximum and the minimum values in the data.

Model Building

Moving on to the model building part, we see there are a lot of variables in this dataset that we cannot deal with. Hence, we’ll use RFE to select a small set of features from this pool. Below is the code for the same:

# Import 'LogisticRegression' and create a LogisticRegression object from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression() # Import RFE and select 15 variables from sklearn.feature_selection import RFE

rfe = RFE(logreg, 15)

rfe = rfe.fit(X_train, y_train)

We’ll now use statsmodels to create a logistic regression models based on p-values and VIFs. To begin with we’ll create a model on the train set after adding a constant and output the summary. This is how the generalized model regression results would look like:

We’ll also compute the VIFs of all features in a similar fashion and drop variables with a high p-value and a high VIF. After re-fitting the model with the new set of features, we’ll once again check for the range in which the p-values and VIFs lie. If appropriate, we’ll proceed with model evaluation as the next step.

Model Evaluation

We’ll now predict the probabilities on the train set and create a new dataframe containing the actual conversion flag and the probabilities predicted by the model. Add a column to capture the predicted values with a condition of being equal to 1 (in case value for paid probability is greater than 0.5) or else 0.

Some important concepts to be familiar with before we begin evaluating the model:

Confusion Matrix

We define classification accuracy as the ratio of correct predictions to total predictions. One of the major issues with this approach is that it often hides the very detail that you might require to better understand the performance of your model. A very likely example where you can encounter this problem is when you’re working with a data having more than 2 classes. You may achieve an accuracy rate of, say 85%, but you’ll not know if this is because some classes are being neglected by your model or whether all of them are being predicted equally well.

Confusion matrix combats this problem. It tells you the exact number of ways your model is confused when it makes predictions.

It is a summary of prediction results on a classification model.

In two-class problems, we construct a confusion matrix by assigning the event row as “positive” and the no-event row as “negative”. The event column of predictions is assigned as “true” and the no-event one as “false”.

The matrix would then consist of the following elements:

(i) True positive — for correctly precited event values

(ii) True negative — for correctly predicted no-event values

(iii) False positive — for incorrectly predicted event values

(iv) False negative — for incorrectly predicted no-event values

Some basic performance measures derived from the confusion matrix are:

(a) Sensitivity: Sensitivity (SN) is calculated as the number of correct positive predictions divided by the total number of positives. It is also called recall (REC) or true positive rate (TPR).

Mathematically,

(b) Specificity: Specificity (SP) is calculated as the number of correct negative predictions divided by the total number of negatives. It is also called true negative rate (TNR).

Mathematically,

(c) Precision: Precision (PREC) is calculated as the number of correct positive predictions divided by the total number of positive predictions. It is also called positive predictive value (PPV)

Mathematically,

(d) Recall: This is the fraction of all existing positives that we predict correctly.

Mathematically,

We’ll use the above matrix and the metrics to evaluate the model. 0.5 was a randomly selected value to test the model performance. We need to optimise the threshold to get better results, which we’ll do by plotting and analysing the ROC curve.

The Receiver Operating Characteristic curve is basically a plot between false positive rate and true positive rate for a number of threshold values lying between 0 and 1.

The ROC curve helps us compare curves of different models with different thresholds whereas the AUC (area under the curve) gives us a summary of the model skill.

Here, the AUC is 0.86 which seems quite good. To find the optimal cut-off point, let’s also check for sensitivity and specificity of the model at different probability cut-offs and plot the same.

At 0.42, the curves of the three metrics seem to intersect and therefore we’ll choose this as our cut-off value. (You may want to calculate the metrics, again, using this point) We’ll make predictions on the test set following the same approach.

Precision-Recall Trade-off

When building a classification model, we need to consider both precision and recall. It is always possible to increase one value at the expense of the other (recall-focussed model/precision-focussed model). In choosing an optimal value for both these metrics, we should always keep in mind the type of problem we are aiming to solve.

Let’s check this trade-off for our chosen value of cut-off (i.e., 0.42).

The trade-off curve and the metrics seem to suggest the cut-off point we have chosen is optimal. So we’ll run one final prediction on our test set and confirm the metrics.

# Let's check the overall accuracy metrics.accuracy_score(y_pred_final['Converted'], y_pred_final.final_predicted) TP = confusion2[1,1] # true positive

TN = confusion2[0,0] # true negatives

FP = confusion2[0,1] # false positives

FN = confusion2[1,0] # false negatives # Calculate Precision TP/(TP+FP) # Calculate Recall TP/(TP+FN)

The metrics seem to hold on the test data. Looks like we have created a decent model as the metrics are decent for both the test and the train datasets.

You are now familiar with the basics of building and evaluating logistic regression models using Python. Generally, it is a straightforward approach:

(i) Import the necessary packages and libraries

(ii) Data cleaning, transformation

(iii) Classification model to be created and trained with the existing data

(iv) Evaluate and check for model performance

(v) Make predictions on the test sets