Logistic Regression in Depth
By: Rakesh Ranjan
Tags: Classification Metrics, Logistic Regression, Cost Function, Python, Machine Learning

Logistic Regression is a classification algorithm. It is used to predict a binary outcome (1 / 0, Yes / No, True / False) given a set of independent variables. To represent binary/categorical outcome, we use dummy variables. You can also think of logistic regression as a special case of linear regression when the outcome variable is categorical, where we are using log of odds as dependent variable. Logistic regression is named for the function used at the core of the method, the logistic function.

The logistic function, also called the sigmoid function was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It’s an S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits.

1 / (1 + e^-value)

Equation For Logistic Regression

Input values (x) are combined linearly using weights or coefficient values (referred to as the Greek capital letter Beta) to predict an output value (y). A key difference from linear regression is that the output value being modeled is a binary values (0 or 1) rather than a numeric value.

Below is an example logistic regression equation:

y = e^(b0 + b1*x) / (1 + e^(b0 + b1*x))

Where y is the predicted output, b0 is the bias or intercept term and b1 is the coefficient for the single input value (x). Each column in your input data has an associated b coefficient (a constant real value) that must be learned from your training data. Below shows a sample logistic regression graph with changing values for coefficients.

Logistic Regression implemented using Sigmoid function

Cost Function

The cost function is minimized for any machine learning algorithm to find the best fitted line or the most effective coefficients. Here we have — b0 + b1*x, and we need to find a cost function that will help us in minimize the coefficients. We will use gradient descent on the cost functions to minimize the given equation and the point where slope is minimum will give us best decision boundary for the logistic regression.

The cost function for Logistic regression can be given as below-

This can be combined as-

Combined cost function with both the cases.

Performance Metrics

Confusion Matrix: Before dealing with the performance metrics let us talk about the confusion matrix first and understand it, a sample confusion matrix is shown below. Y-axis represent the actual labels and X-axis represent the predicted labels. We use this to calculate performance metrics as discussed below.

2. Accuracy: Accuracy represents the number of correctly classified data instances over the total number of data instances.

3. Precision: Precision should ideally be 1 (high) for a good classifier. Precision becomes 1 only when the numerator and denominator are equal i.e TP = TP +FP, this also means FP is zero. As FP increases the value of denominator becomes greater than the numerator and precision value decreases

4. Recall: Recall is also known as sensitivity or true positive rate. Recall should ideally be 1 (high) for a good classifier. Recall becomes 1 only when the numerator and denominator are equal i.e TP = TP +FN, this also means FN is zero. As FN increases the value of denominator becomes greater than the numerator and recall value decreases

5. F1 Score: F1-score is a metric which takes into account both precision and recall. F1 Score becomes 1 only when precision and recall are both 1. F1 score becomes high only when both precision and recall are high. F1 score is the harmonic mean of precision and recall and is a better measure than accuracy.

Summary

Logistic Regression is one of the most widely used industry standard Machine Learning Algorithm both because of it’s performance and ease of interpretability. What features are important for business can be determined by their coefficients when we look at the final model that is generated and can be used to drive business decisions.