Logistic Regression under Classification Algorithms in ML
By: Rukshan Pramoditha
Tags: Logistic Regression, Machine Learning, Data Science, Supervised Learning, Classification

Classification in Machine Learning

Classification is a process of categorizing a given set of data into classes. The classes are often referred to as targets, labels or categories. A classification problem with two classes is known as a two-class classification problem. Those with more than two classes are known as multi-class classification problems.

The outcome of a classification problem is a discrete value (known class). The outcome of a classification problem can also be a continuous value, (e.g. likelihood of an observation belonging to a particular class), which can be converted into a relevant class as in “win percentage of candidate A = 0.65 → win class”.

Classification techniques play an important role in machine learning applications. Most of the problems in Data Science are classification problems. Logistic Regression, the algorithm I discuss here, can be used for various classification problems such as spam emails detection. The main goal is to identify which class or category (“spam” or “not spam”) the new data (new email) will fall into.

Classification Terminologies in Machine Learning

Classifier: An algorithm that is used to map the input data to a specific category

An algorithm that is used to map the input data to a specific category Classification Model: This will predict the class or category for the new data

This will predict the class or category for the new data Feature: An individual measurable property

An individual measurable property Train the Classifier: Each classifier in scikit-learn uses a fit (x, y) method to fit the model for the given train data x and label y.

Types of Learners in Classification

There are two types of learners (machine learning algorithms) in classification:

Lazy Learners

Lazy learners simply store the training data & the construction of the classification model is delayed until a testing data appears in the system. When it does, classification is conducted based on the stored training data. Because of that, lazy learners have less training time but requires more time in predicting. Lazy learning is useful for large & continuously changing datasets.

E.g. K-Nearest Neighbour (K-NN)

Eager Learners

Eager learners construct a classification model based on the given training data before receiving data for classification. Because of that, eager learners take a long time to train and less time to predict.

E.g. Decision Tree, Naive Bayes

Logistic Regression

Logistic regression is a statistical method which uses categorical and continuous variables to predict a categorical outcome. When the dependent variable (outcome variable) of choice has two categorical outcomes, the analysis is termed binary logistic regression. However, if the outcome variable consists of more than two levels, the analysis is referred to as multinomial logistic regression.

Binary Logistic Regression: The dependent variable has only two categorical outcomes.

The dependent variable has only two categorical outcomes. Multinomial Logistic Regression: The dependent variable has more than two levels.

When predicting a binary outcome, we do not have a linear relationship between the features and the outcome variable; an assumption of linear regression. Thus, to express a nonlinear relationship in a linear way, we must transform the data using logarithmic transformation. As a result, logistic regression allows us to predict the probability of the binary outcome occurring given the feature(s) in the model. The output (dependent variable) is a probability ranging from 0 (not going to happen) to 1 (definitely will happen), or a categorization that says something is either part of the category or not part of the category (E.g. spam or not spam).

Now, we can consider the following example to understand the use of logistic regression.

E.g. I have a dataset, as in the following format, containing the information about voter income and voting preference. For this dataset, I can see that low-income voters tend to vote for candidate B, while high-income voters tend to vote for candidate A.

Format of the dataset

With the actual dataset, I am very interested to answer for a Data Science Question such as For which candidate will voters give votes based on their income level?

This problem has binary outcomes as shown in the following chart.

If I applied Linear Regression to this problem, it would lead to strange values as in the following chart. Consider the case of a very low-income voter (near to 0), and you can see from the chart that the predicted result is a negative value which is strange.

The following chart shows how logistic regression solves this problem. Instead of drawing a straight line cutting through the points, you now use a curved line to try to fit all of the points on the chart.

By looking at the chart, you can see Logistic Regression predicts the probability of an outcome, rather than a specific value. The output will be a value from 0 to 1, where anything less than (or equal to) 0.5 (known as the threshold) will be considered as voting for candidate B, and anything greater than 0.5 will be considered as voting for candidate A.

How the Logistic Regression Algorithm Works?

Understanding Odds

Before we discuss the details of the logistic regression algorithm, we first need to discuss one important term — odds. Odds are defined as the ratio of the probability of success to the probability of failure.

Calculating the odds of an event happening

For example, the odds of landing the head when you flip a coin are 1. This is because you have a 0.5 probability of landing the head and a 0.5 probability of landing the tail.

When you flip a coin which was rigged in such a way that the probability of landing the head is 0.8 and the probability of landing the tail is 0.2, then the odds of landing the head is 0.8/0.2 = 4. That is, you are 4 times more likely to land the head than the tail. Likewise, the odds of getting the tail are 0.2/0.8 = 0.25.

Logit Function

When you apply the natural logarithm function to the odds, you get the logit function. The logit function is the logarithm of the odds.

The formula for the logit function

The logit function transfers a variable on (0, 1) into a new variable on (–∞, ∞). To see this relationship, you can use the following code snippet & its output produced using the Jupyter Notebook. ( The Jupyter Notebook is an open-source web application which is used to do data science projects in a modern way. It allows you to create easily sharable documents which contain Python Code, Equations, Visualizations and Markup Text together. Later, I will write a separate article that describes the fundamentals of the Jupyter Notebook.)

The logit curve plotted using matplotlib

Sigmoid Curve

For the logit curve, observe that the x-axis is the probability and the y-axis is the real-number range. For logistic regression, what you really want is a function that maps numbers on the real number system to the probabilities, which is exactly what you get when you flip the axes of the logit curve.

Flipping the logit curve into a Sigmoid curve

When you flip the axes, the curve that you get is called the sigmoid curve. The sigmoid curve is obtained using the Sigmoid function, which is the inverse of the logit function. The Sigmoid function is used to transform values on (–∞,∞) into numbers on (0, 1).

The formula for the Sigmoid function

To see how the sigmoid curve is obtained, you can use the following code snippet & its output produced using the Jupyter Notebook.

The Sigmoid curve plotted using matplotlib

Replacing L by (α+βx)

Note that L of the Sigmoid function can be replaced by (α+βx) when there is 1 predictor (explanatory variable/independent variable).

P is the probability of the outcome occurring.

is the probability of the outcome occurring. e is the base of the natural logarithm.

is the base of the natural logarithm. x is the value of the predictor

is the value of the predictor α is the intercept.

is the intercept. β is the coefficient of the predictor x.

α and β are unknown, and they must be estimated based on the available training data using a technique known as Maximum Likelihood Estimation (MLE).

Note that the L of the Sigmoid function can also be replaced by:

(α + β1x1 + β2x2 + … + βnXn) when there is n number of predictors.

Assumptions

The outcome variable has only levels (2 or more than 2), but not numerical values.

The data is free of missing values.

The predictors (x-variables) are independent of each other.

Note: The implementation of the Logistic Regression with Python will be discussed later with real-world examples after discussing the Python programming fundamentals, Matplotlib & Scikit-learn.

What is next?

Next time, I will discuss another machine learning algorithm called K-Nearest Neighbours (K-NN) Algorithm. The progress is shown in the following algorithm chart.