Regression for Classification | Hands on Experience
By: Yasas Sandeepa
Tags: Regression, Hands On Tutorials, Python, Classification, Machine Learning

Regression for Classification | Hands on Experience

Photo by Lance Anderson on Unsplash

We all have developed numerous regression models in our lives. But only few are familiar with using regression models for classification. So my intention is to reveal the beauty of this hidden world.

As we all know, when we want to predict a continuous dependent variable from a number of independent variables, we used linear/polynomial regression. But when it comes to classification, we can’t use that anymore.

Fundamentally, classification is about predicting a label and regression is about predicting a quantity.

Why linear regression can’t use for classification? The main reason for that is the predicted values are continuous, not probabilistic. So we can’t get an exact class to accomplish the classification. You will further understand by glancing at the below predictions.

Image by author

Probability is ranged between 0 and 1. But in linear regression, we are predicting an absolute number, which can range outside 0 and 1.

Yes, you can still normalize the value to the 0–1 range but the results may be worse. This is because Linear Regression fit is highly affected by the inclusion of an outlier. Even a small outlier will ruin your classification.

On the other hand, using linear regression for multi class prediction makes no sense. Linear regression assumes an order between 0, 1, and 2, whereas in the classification regime these numbers are mere categorical placeholders.

To overcome the aforementioned problem, there are 2 great solutions.

Logistic Regression — For binary classification Softmax Regression — For multi class classification

I am using Red Wine Quality Dataset (in Kaggle) for demonstrating this to you.

Original data set is publicly available in the UCI machine learning repository.

Note: I will not perform any detailed preprocessing or dimensional reduction techniques today as my intention is to walk through you mostly on the classification models.

Okay! First, we will see how a binary classification is achieved using Logistic Regression.

Logistic Regression

Before applying the model, let’s understand some of the core concepts in logistic regression.

I’ll show you how logistic regression works with an example. Consider 2 outcome probabilities of tossing a coin.

Image by author

Let’s take the probability of displaying head as p and probability of displaying tail as q. Then we define a new concept called Odds as p/q.

In this scenario, we have only two possibilities. That is either Head or Tail. So we can write q as 1-p. Now we will see the variation of p. It can vary up to 0 to 1. Let’s take 3 fair points and analyze them with Odds.

Image by author

Now we can get the log of that Odds. Then it will be known as Log Odds or Logit [log(p/q)]. These log-odds hold the value condition only for 2 classes.

Image by author

As you can see, it is showing a symmetric distribution. When p=0.5, the Logit function does not favor anyone. When p=1, it is favored by +inf, and at the same time, q (p-1) is favored by -inf. This characteristic is the intuition for the Logistic Regression as it can classify a binary problem with its nature.

Image by author

Then we can derive the Logistic Function from this Logit Function.

Image by author

If we plot the graph, it will be viewed as follows.

Image by author

The following diagram briefly shows how Logistic regression works.

Image by author

You can dig deeper into segments like analyzing the loss function (Binary cross-entropy) and formulating the decision boundary. It is out of scope from this article as it takes more time to explain. Let’s see the implementations without any further due.

First of all let’s load our CSV file to a pandas Dataframe.