Quick and Easy Explanation of Logistic Regression | by Renu Khandelwal | Towards Data Science

It is a predictive algorithm using independent variables to predict the dependent variable, just like Linear Regression, but with a difference that the dependent variable should be categorical variable.
Independent variables can be numeric or categorical variables, but the dependent variable will always be categorical
Logistic regression is a statistical model that uses Logistic function to model the conditional probability.
For binary regression, we calculate the conditional probability of the dependent variable Y, given independent variable X
It can be written as P(Y=1|X) or P(Y=0|X)
This is read as the conditional probability of Y=1, given X or conditional probability of Y=0, given X.
P(Y |X) is approximated as a sigmoid function applied to a linear combination of input features
An example of logistic regression can be to find if a person will default their credit card payment or not. The probability of a person defaulting their credit card payment can be based on the pending credit card balance and income etc.
hence, we can write P(default=yes|balance)
when the P(default=yes)≥0.5, then we say the person will default their payment.
When the P(default=yes)<0.4, then we say the person will NOT default their payment.
The probability will always range between 0 and 1. In the case of binary classification, the probability of defaulting payment and not defaulting payment will sum up to 1
Logistic Regression can be used for binary classification or multi-class classification.
Binary classification is when we have two possible outcomes like a person is infected with COVID-19 or is not infected with COVID-19. In multi-class classification, we have multiple outcomes like the person may have the flu or an allergy, or cold or COVID-19.
No outliers in the data. An outlier can be identified by analyzing the independent variables
No correlation (multi-collinearity) between the independent variables.
Logistic regression uses logit function, also referred to as log-odds; it is the logarithm of odds. The odds ratio is the ratio of odds of an event A in the presence of the event B and the odds of event A in the absence of event B.
P is the probability that event Y occurs. P(Y=1)
P/(1-P) is the odds ratio
θ is a parameters of length m
Logit function estimates probabilities between 0 and 1, and hence logistic regression is a non-linear transformation that looks like S- function shown below.
The parameter “θ” of the logistic function can be estimated using the maximum likelihood estimation(MLE) framework. MLE searches for the parameters that best fit the joint probability if the independent variables X.
MLE will give us values for parameter “θ” that would maximize the probability close to 1 for the person who would default their payment and a number close to 0 for all individuals who would not default their payment.
A confusion matrix is a table that tells us how many actual values and predicted values exist for different classes predicted by the model. Also referred to as the Error matrix.
Explaining the key terms
True Positive: When a female is pregnant, and we predicted that she is pregnant
True Negative: When a male is not pregnant, and we predicted that he is not pregnant
False Negative: When a female is pregnant, but we predict that she is not pregnant. Also called Type 2 error.
False Positive: A male cannot be pregnant, but we predict he is pregnant. Also called Type 1 error
Data can be downloaded from here
Importing required libraries
import pandas as pdimport numpy as npfrom matplotlib import pyplot as pltimport seaborn as snsfrom sklearn.model_selection import train_test_split%matplotlib inline
Reading the data
Download the data from the link and save it in a CSV file and add the headers.
dataset=pd.read_csv("iris.csv")dataset.info()
displaying a few rows from the dataset
dataset.head(5)
Data analysis using visualization
Visualize the data to get a better understanding of the independent and dependent variables
sns.boxplot(x="class",y="petal length",data=dataset)plt.show()
Scatterplot of different classes of iris using petal length and petal width
sns.scatterplot(x='petal length', y='petal width', hue="class", data=dataset)
Scatterplot of different classes of iris using sepal length and sepal width
sns.scatterplot(x='sepal length', y='sepal width', hue="class", data=dataset)
Encode the class of the iris from categorical to a numeric value
The target variable is a string representing the class of the iris plant. We encode the categorical string data to the numeric value for the model to understand
from sklearn.preprocessing import LabelEncoderle= LabelEncoder()y_encoded=le.fit_transform(y)
Printing the encoded value for the target variable.
y_encoded
We have a multi-class classification problem as we have three distinct values for the class of iris plant
Split dataset into train and test dataset
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test= train_test_split(X,y_encoded,test_size=0.2)
We are training the dataset for multi-class classification using logistic regression
from sklearn.linear_model import LogisticRegressionclf = LogisticRegression(random_state=0).fit(X_train, y_train)
Predict the class of the iris for the test data
y_pred=clf.predict(X_test)
Evaluate the performance of the Logistic Regression Model
Use the confusion matrix to evaluate the performance for the multi-class classification
from sklearn.metrics import confusion_matrixcnf_matrix=confusion_matrix(y_test, y_pred, labels=[0,1,2])cnf_matrix
Ths output does not help much, so we inverse transform the numeric target variable back to the original class name
y_pred_orig= le.inverse_transform(y_pred)
We also write a function to display the confusion matrix in a more readable format
def plot_confusion_matrix(cm, classes,                          title='Confusion Matrix',                          cmap=plt.cm.Greens):       import itertools        print('Confusion Matrix')    plt.imshow(cm, interpolation='nearest', cmap=cmap)    plt.title(title)    tick_marks = np.arange(len(classes))    plt.xticks(tick_marks, classes, rotation=90)    plt.yticks(tick_marks, classes)    fmt = '.2f'     thresh = cm.max() / 2.    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):        plt.text(j, i, format(cm[i, j], fmt),                 horizontalalignment="center",                 color="white" if cm[i, j] > thresh else "black")plt.ylabel('Actual label')    plt.xlabel('Predicted label')    plt.tight_layout()
Displaying the confusion matrix
plt.figure()plot_confusion_matrix(cnf_matrix, classes=le.classes_,                      title='Confusion matrixfor Iris test data')
Logistic regression is used for binary or multi-class classification, and the target variable always has to be categorical.