Machine Learning Basics: Logistic Regression | by Gurucharan M K | Towards Data Science

In the previous stories, I had given an explanation of the program for implementation of various Regression models. As we move on to Classification, isn’t it surprising as to why the title of this algorithm still has the name, Regression. Let us understand the mechanism of the Logistic Regression and learn to build a classification model with an example.
Logistic Regression is a classification model that is used when the dependent variable (output) is in the binary format such as 0 (False) or 1 (True). Examples include such as predicting if there is a tumor (1) or not (0) and if an email is a spam (1) or not (0).
The logistic function, also called as sigmoid function was initially used by statisticians to describe properties of population growth in ecology. The sigmoid function is a mathematical function used to map the predicted values to probabilities. Logistic Regression has an S-shaped curve and can take values between 0 and 1 but never exactly at those limits. It has the formula of 1 / (1 + e^-value).
Logistic Regression is an extension of the Linear Regression model. Let us understand this with a simple example. If we want to classify if an email is a spam or not, if we apply a Linear Regression model, we would get only continuous values between 0 and 1 such as 0.4, 0.7 etc. On the other hand, the Logistic Regression extends this linear regression model by setting a threshold at 0.5, hence the data point will be classified as spam if the output value is greater than 0.5 and not spam if the output value is lesser than 0.5.
In this way, we can use Logistic Regression to classification problems and get accurate predictions.
To apply the Logistic Regression model in practical usage, let us consider a DMV Test dataset which consists of three columns. The first two columns consist of the two DMV written tests (DMV_Test_1 and DMV_Test_2) which are the independent variables and the last column consists of the dependent variable, Results which denote that the driver has got the license (1) or not (0).
In this, we have to build a Logistic Regression model using this data to predict if a driver who has taken the two DMV written tests will get the license or not using those marks obtained in their written tests and classify the results.
As always, the first step will always include importing the libraries which are the NumPy, Pandas and the Matplotlib.
import numpy as npimport matplotlib.pyplot as pltimport pandas as pd
In this step, we shall get the dataset from my GitHub repository as “DMVWrittenTests.csv”. The variable X will store the two “DMV Tests ”and the variable Y will store the final output as “Results”. The dataset.head(5)is used to visualize the first 5 rows of the data.
dataset = pd.read_csv('https://raw.githubusercontent.com/mk-gurucharan/Classification/master/DMVWrittenTests.csv')X = dataset.iloc[:, [0, 1]].valuesy = dataset.iloc[:, 2].valuesdataset.head(5)>>DMV_Test_1   DMV_Test_2   Results34.623660    78.024693    030.286711    43.894998    035.847409    72.902198    060.182599    86.308552    179.032736    75.344376    1
In this step, we have to split the dataset into the Training set, on which the Logistic Regression model will be trained and the Test set, on which the trained model will be applied to classify the results. In this the test_size=0.25 denotes that 25% of the data will be kept as the Test set and the remaining 75% will be used for training as the Training set.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)
This is an additional step that is used to normalize the data within a particular range. It also aids in speeding up the calculations. As the data is widely varying, we use this function to limit the range of the data within a small limit ( -2,2). For example, the score 62.0730638 is normalized to -0.21231162 and the score 96.51142588 is normalized to 1.55187648. In this way, the scores of X_train and X_test are normalized to a smaller range.
from sklearn.preprocessing import StandardScalersc = StandardScaler()X_train = sc.fit_transform(X_train)X_test = sc.transform(X_test)
In this step, the class LogisticRegression is imported and is assigned to the variable “classifier”. The classifier.fit() function is fitted with X_train and Y_train on which the model will be trained.
from sklearn.linear_model import LogisticRegressionclassifier = LogisticRegression()classifier.fit(X_train, y_train)
In this step, the classifier.predict() function is used to predict the values for the Test set and the values are stored to the variable y_pred.
y_pred = classifier.predict(X_test) y_pred
This is a step that is mostly used in classification techniques. In this, we see the Accuracy of the trained model and plot the confusion matrix.
The confusion matrix is a table that is used to show the number of correct and incorrect predictions on a classification problem when the real values of the Test Set are known. It is of the format
The True values are the number of correct predictions made.
from sklearn.metrics import confusion_matrixcm = confusion_matrix(y_test, y_pred)from sklearn.metrics import accuracy_score print ("Accuracy : ", accuracy_score(y_test, y_pred))cm>>Accuracy :  0.88>>array([[11,  0],       [ 3, 11]])
From the above confusion matrix, we infer that, out of 25 test set data, 22 were correctly classified and 3 were incorrectly classified. Pretty good for a start, isn’t it?
In this step, a Pandas DataFrame is created to compare the classified values of both the original Test set (y_test) and the predicted results (y_pred).
df = pd.DataFrame({'Real Values':y_test, 'Predicted Values':y_pred})df>> Real Values   Predicted Values1             10             00             00             01             11             11             01             10             01             10             00             00             01             11             01             10             01             11             01             10             00             01             11             10             0
Though this visualization may not be of much use as it was with Regression, from this, we can see that the model is able to classify the test set values with a decent accuracy of 88% as calculated above.
In this last step, we visualize the results of the Logistic Regression model on a graph that is plotted along with the two regions.
from matplotlib.colors import ListedColormapX_set, y_set = X_test, y_testX1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),             alpha = 0.75, cmap = ListedColormap(('red', 'green')))plt.xlim(X1.min(), X1.max())plt.ylim(X2.min(), X2.max())for i, j in enumerate(np.unique(y_set)):    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],                c = ListedColormap(('red', 'green'))(i), label = j)plt.title('Logistic Regression')plt.xlabel('DMV_Test_1')plt.ylabel('DMV_Test_2')plt.legend()plt.show()
In this graph, the value 1 (i.e, Yes) is plotted in “Red” color and the value 0 (i.e, No) is plotted in “Green” color. The Logistic Regression line separates the two regions. Thus, any data with the two data points (DMV_Test_1 and DMV_Test_2) given, can be plotted on the graph and depending upon which region if falls in, the result (Getting the Driver’s License) can be classified as Yes or No.
As calculated above, we can see that there are three values in the test set that are wrongly classified as “No” as they are on the other side of the line.
Thus in this story, we have successfully been able to build a Logistic Regression model that is able to predict if a person is able to get the driving license from their written examinations and visualize the results.
I am also attaching the link to my GitHub repository where you can download this Google Colab notebook and the data files for your reference.
github.com
You can also find the explanation of the program for other Classification models below:
Logistic Regression
K-Nearest Neighbors (KNN) Classification (Coming Soon)
Support Vector Machine (SVM) Classification (Coming Soon)
Naive Bayes Classification (Coming Soon)
Random Forest Classification (Coming Soon)
We will come across the more complex models of Regression, Classification and Clustering in the upcoming articles. Till then, Happy Machine Learning!