Logistic Regression Algorithm using Data Mining WEKA TOOL
By: Siddharth Singh
Tags: Weka, Machine Learning, Logistic Regression

Working Principle

Logistic Regression is a classification algorithm. It is a predictive modeling algorithm that is used when the dependent variable(target) is categorical in nature.

Logistic Regression is most commonly used when the data in question has binary output, so when it belongs to one class or another, or is either a 0 or 1. These types of problems are known as Binary Classification problems. When using logistic regression, a threshold is usually specified that indicates at what value the example will be put into one class vs. the other class.

Although, sometime we come across more than 2 classes and still it is a classification problem. These types of problems are known as Multi class classification problems. We can accomplish this by applying a “one vs rest” strategy.

Why not use Linear Regression?

Suppose we have a data of tumor size vs its malignancy. As it is a classification problem, if we plot, we can see, all the values will lie on 0 and 1. And if we fit best found regression line, by assuming the threshold at 0.5, we can do line pretty reasonable job.

We can decide the point on the x axis from where all the values lie to its left side are considered as negative class and all the values lie to its right side are positive class.

But what if there is an outlier in the data. Things would get pretty messy. For example, for 0.5 threshold,

If we fit best found regression line, it still won’t be enough to decide any point by which we can differentiate classes. It will put some positive class examples into negative class. The green dotted line (Decision Boundary) is dividing malignant tumors from benign tumors but the line should have been at a yellow line which is clearly dividing the positive and negative examples. So just a single outlier is disturbing the whole linear regression predictions. And that is where logistic regression comes into a picture.

Sigmoid Function

To deal with outliers, Logistic Regression uses Sigmoid function.

The Sigmoid function/Logistic function, it is an S-shaped when plotted on a graph, that gets closer to 1 as the value of input variable increases above 0 and gets closer to 0 as the input variable decreases below 0. The output of the sigmoid function is 0.5 when the input variable is 0.

Thus, if the output is more than 0.5, we can classify the outcome as 1 (or positive) and if it is less than or equal to 0.5, we can classify it as 0 (or negative).

It is defined as:

And if we plot it, the graph will be S curve,

Let’s consider t as linear function in a univariate regression model.

So the Logistic Equation will become

Now, when logistic regression model come across an outlier, it will take care of it.

But sometime it will shift its y axis to left or right depending on outliers positions.

What is Decision Boundary?

Decision boundary helps to differentiate probabilities into positive class and negative class.

Linear Decision Boundary

Non Linear Decision Boundary

One vs Rest Strategy for Multiclass Classification

Let’s say you have three different classes that instances in your dataset could fall into, and if you had these three classes you could treat them as three different binary classification problems.

For instance, you would train a classifier on just the examples belonging to Class A vs. all the examples belonging to all other classes. You would then do the same thing for Class B, and finally for Class C.

After the classifiers have learned to distinguish their chosen class from other classes, you just run the three classifiers on the inputs and whichever classifier is most confident it has chosen the correct class for that example that is what class you put the example in.

Implementation in Weka

Parameters:

Step 1 Click on Classification Tab.

Step 2 Click on choose button.

Step 3 Open function folder and select Logistic.

Step 4 Click on percentage spit and change it to 80% and click start.

Result :-