Perfect Recipe for Classification Using Logistic Regression
By: Ashwin Raj
Tags: Supervised Learning, Towards Data Science, Logistic Regression, Machine Learning, Classification

Perfect Recipe for Classification Using Logistic Regression

Supervised Learning is an essential part of Machine Learning. Classification techniques are used when the variable to be predicted is categorical. A common example of classification problem is trying to classify an Iris flower among its three different species.

Logistic regression is a classification technique borrowed by machine learning from the field of statistics. Logistic Regression is a statistical method for analyzing a dataset in which there are one or more independent variables that determine an outcome. The intention behind using logistic regression is to find the best fitting model to describe the relationship between the dependent and the independent variable.

In this article, we will first be taking a theoretical approach on what Logistic regression actually is and after that we will be building our very first classification model. But before getting started with this article, I would suggest you to take a look on my previous article to get a gist of what classification problem actually are, in virtue of Machine Learning

Why not Linear Regression:

Consider a scenario wherein we need to classify whether a particular type of cancer is malignant or not. If we use linear regression for this problem, there is a need for setting up a threshold based on which classification can be done as linear regression return continous value.

Image from Gfycat

But in scenarios wherein the actual class is malignant, but the predicted continuous value is 0.4. Given that the threshold value is 0.5, the data point will be classified as not malignant which may lead to serious consequence. As a result it can be inferred that linear regression is not suitable for classification problems as it is unbounded and the predicted value is continuous, and not probabilistic.

The decision for converting a predicted probability into a class label is decided by the parameter known as Threshold. A value above that threshold indicates one class while the one below indicates the other.

Image from Freepik

Not only this but linear regression is also highly sensitive to imbalanced data as it tries to fit the line by minimising the error(distance between the line and the actual value). As a result it can be inferred that Linear regression is better off with regression problems and is not suitable for classification problems, which brings logistic regression into picture.

Logistic Regression:

Logistic Regression is a classification technique used in machine learning. It uses a logistic function to model the dependent variable. The dependent variable is dichotomous in nature, i.e. there could only be two possible classes (eg.: either the cancer is malignant or not). As a result, this technique is used while dealing with binary data.

Image from Gfycat

Types of Logistic Regression:

Though generally used for predicting binary target variables, logistic regression can be extended and further classified into three different types that are as mentioned below:

Binomial: Where the target variable can have only two possible types. eg.: Predicting a mail as spam or not. Multinomial: Where the target variable have three or more possible types, which may not have any quantitative significance. eg.: Predicting disease. Ordinal: Where the target variables have ordered categories. eg.: Web Series ratings from 1 to 5.

In logistic regression in order to map the predicted values to probabilities, sigmoid function is used. This function maps any real value into another value between 0 to 1. This function has a non-negative derivative at each point and exactly one inflection point.

Image from Freepik

Cost Function

A cost function is a mathematical formula used to quantify the error between the predicted values and the expected values. Put simply, a cost function is a measure of how wrong the model is in terms of its ability to estimate the relationship between x and y. The value returned by the cost function is referred to as cost or loss or simply, error. For logistic regression, the cost function is given by the equation:

This negative function is because when we train, we need to maximize the probability by minimizing loss function. Decreasing the cost will increase the maximum likelihood, assuming that samples are drawn from an identically independent distribution.

Now that you have an idea of what Logistic Regression is, we shall now be building our very own logistic regression model. The code and other resources for this classification model can be found here.

Step 1: Importing the Required Libraries

Our first step is to import the libraries required to build our model. It is not necessary to import all the libraries at just one place. To get started we will be importing the Pandas and Numpy libraries.

Image from Freepik

Once these libraries have been imported our next step will be fetching the dataset and loading the data into our notebook. The dataset we’ll be using in this example is about Heart Diseases.

#Import the Libraries and read the data into a Pandas DataFrame import pandas as pd

import numpy as np df = pd.read_csv("framingham_heart_disease.csv")

df.head()

The pandas library is used for accessing the data. The read_csv function inputs the data of the format .csv into the pandas dataframe.

The head() function is used to display first few records of the dataframe. The function by default displays the first five records but can be made to display any number of records by entering the desired value.

Step 2: Cleaning the Dataset

Data cleansing or data cleaning is the process of detecting and correcting corrupt or inaccurate records from a table, or database by identifying incomplete, incorrect or irrelevant parts of the data and then replacing, modifying, or deleting the coarse data.

Image from Freepik

The missing values of the dataset can be detected using the isnull function. These records with missing values are either deleted or filled with the mean value of the record. However, inputting the approximation may introduce variance and/or bias in the model.

The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs and thus result in underfitting.

#Handling missing data series = pd.isnull(df['cigsPerDay'])

On the other hand, variance is an error caused due to the sensitivity of the model towards small fluctuations in the training set. High variance can cause an algorithm to model these fluctuations in the training data, thus resulting in the overfitting of the model.

Image by Author

In this example we also note that certain features of the dataset such as Education does not play a factor in deciding the output. Such features shall be dropped before building the model.

#Dropping unwanted columns data = df.drop(['currentSmoker','education'], axis = 'columns')

cigs = data['cigsPerDay'] cig = cigs.mean()

integer_value = math.floor(cig)

cigs.fillna(integer_value, inplace = True)

data.dropna( axis = 0, inplace = True)

Certain features like CigsPerDay already denotes that the person is a smoker and hence the column CurrentSmoker is of effectively no use.

Step 3: Analyzing the Dataset

Now that our dataset is clean and free of irregularities, the next step will be to create two separate dataframes for people with high risk of heart disease and for those with lower risk of heart disease.

This step can also be referred to as Feature Engineering. Another important step is visualising the data as it greatly helps in determinig which features to select that may yield the most optimum results..

#Analyzing The Dataset Heart_Attack = data[data.TenYearCHD == 1]

No_Heart_Attack = data[data.TenYearCHD == 0] final = data.drop(['diaBP','BMI','heartRate'], axis = 'columns')

Another important step in feature engineering is to scale the data as logistic regression is sensitive to whether or not the data has been scaled. If the data is not scaled, the model might consider 3000 grams to be greater than 5 KG which may result in erroneous prediction.

Image from Freepik

In this example I have also dropped the values for ‘diaBP’, ‘BMI’ and ‘heartRate’ as they have similar values for either values of TenYearCHD.

Step 4: Preparing the Model

Now that we have cleaned the data and selected the effective features, we are all set to fit the model on our training data. For doing this, we first need to split the dataset into training and testing data with a given random state, so that the output remains same every time the program is executed. In this example, the random state is 99.

#Preparing the model from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.20, random_state = 99) from sklearn.linear_model import LogisticRegression

model = LogisticRegression() model.fit(X_train, y_train)

model.score(X_test,y_test)

Once done, import LogisticRegression from sklearn.linear_model and fit the regressor over the training data. The performance of the model can be evaluated by calculating the model score.

In this example, the model is found to be 85.6% accurate. Confusion Matrix is also a good technique for summarizing the performance of the prediction model. In it there are two possible predicted classes - Positive and Negative.

Image by Author

The confusion matrix can be further used for determining various important metrics including Accuracy, ROC Score, Precision, F Score etc.

Advantages of Logistic Regression

Logistic Regression is one of the most efficient technique for solving classification problems. Some of the advantages of using Logistic regression are as mentioned below.

Logistic regression is easier to implement, interpret, and very efficient to train. It is very fast at classifying unknown records. It performs well when the dataset is linearly separable. It can interpret model coefficients as indicators of feature importance.

Image by Gfycat

Disadvantages of Logistic Regression

Though used widely, Logistic Regression also comes with some limitations that are as mentioned below:

It constructs linear boundaries. Logistic Regression needs that independent variables are linearly related to the log odds. The major limitation of Logistic Regression is the assumption of linearity between the dependent variable and the independent variables. More powerful and compact algorithms such as Neural Networks can easily outperform this algorithm.

Summarizing What You Learned

To summarise what we have learned in this article, first we discussed about what Logistic Regression is and how it can be used to build classification models. We then discussed about the various different types of Logistic Regression, its cost function and the metrics that are used to evaluate the performance of the classification model.

We then took a hands on experience of the algorithm where we built our very own classification model. To prop up our learning we also discussed about the advantages and disadvantages of this technique.

With that, we have reached the end of this article. I hope this article would have helped you get a solid understanding of what Logistic Regression is and when to use it in your Machine Learning journey. If you have any question or if you believe I have made any mistake, feel free to contact me! You can get in touch with me via: E-Mail or LinkedIn. Happy Learning!