Logistic Regression-Theory and Practice | by Gokul S Kumar | Towards Data Science

In this article, I will be explaining how to use the concept of regression, in specific logistic regression to the problems involving classification. Classification problems are everywhere around us, the classic ones would include mail classification, weather classification, etc. All these data, if needed can be used to train a Logistic regression model to predict the class of any future example.
This article is going to cover the following sub-topics:
Introduction to classification problems.Logistic regression and all its properties such as hypothesis, decision boundary, cost, cost function, gradient descent, and its necessary analysis.Developing a logistic regression model from scratch using python, pandas, matplotlib, and seaborn and training it on the Breast cancer dataset.Training an in-built Logistic regression model from sklearn using the Breast cancer dataset to verify the previous model.
Introduction to classification problems.
Logistic regression and all its properties such as hypothesis, decision boundary, cost, cost function, gradient descent, and its necessary analysis.
Developing a logistic regression model from scratch using python, pandas, matplotlib, and seaborn and training it on the Breast cancer dataset.
Training an in-built Logistic regression model from sklearn using the Breast cancer dataset to verify the previous model.
Classification problems can be explained based on the Breast Cancer dataset where there are two types of tumors (Benign and Malignant). It can be represented as:
where
This is a classification problem with 2 classes, 0 & 1. Generally, the classification problems have multiple classes say, 0,1,2 and 3.
The link to the Breast cancer dataset used in this article is given below:
www.kaggle.com
Let’s import the dataset to a pandas dataframe:
Let’s import the dataset to a pandas dataframe:
import pandas as pdread_df = pd.read_csv('breast_cancer.csv')df = read_df.copy()
2. The following dataframe is obtained:
df.head()
df.info()<class 'pandas.core.frame.DataFrame'>RangeIndex: 569 entries, 0 to 568Data columns (total 33 columns): #   Column                   Non-Null Count  Dtype  ---  ------                   --------------  -----   0   id                       569 non-null    int64   1   diagnosis                569 non-null    object  2   radius_mean              569 non-null    float64 3   texture_mean             569 non-null    float64 4   perimeter_mean           569 non-null    float64 5   area_mean                569 non-null    float64 6   smoothness_mean          569 non-null    float64 7   compactness_mean         569 non-null    float64 8   concavity_mean           569 non-null    float64 9   concave points_mean      569 non-null    float64 10  symmetry_mean            569 non-null    float64 11  fractal_dimension_mean   569 non-null    float64 12  radius_se                569 non-null    float64 13  texture_se               569 non-null    float64 14  perimeter_se             569 non-null    float64 15  area_se                  569 non-null    float64 16  smoothness_se            569 non-null    float64 17  compactness_se           569 non-null    float64 18  concavity_se             569 non-null    float64 19  concave points_se        569 non-null    float64 20  symmetry_se              569 non-null    float64 21  fractal_dimension_se     569 non-null    float64 22  radius_worst             569 non-null    float64 23  texture_worst            569 non-null    float64 24  perimeter_worst          569 non-null    float64 25  area_worst               569 non-null    float64 26  smoothness_worst         569 non-null    float64 27  compactness_worst        569 non-null    float64 28  concavity_worst          569 non-null    float64 29  concave points_worst     569 non-null    float64 30  symmetry_worst           569 non-null    float64 31  fractal_dimension_worst  569 non-null    float64 32  Unnamed: 32              0 non-null      float64dtypes: float64(31), int64(1), object(1)memory usage: 146.8+ KB
Let us plot the mean area of the clump and its classification and see if we can find a relation between them.
import matplotlib.pyplot as pltimport seaborn as snsfrom sklearn import preprocessinglabel_encoder = preprocessing.LabelEncoder()df.diagnosis = label_encoder.fit_transform(df.diagnosis)sns.set(style = 'whitegrid')sns.lmplot(x = 'area_mean', y = 'diagnosis', data = df, height = 10, aspect = 1.5, y_jitter = 0.1)
We can infer from the plot that most of the tumors having an area less than 500 are benign(represented by zero) and those having area more than 1000 are malignant(represented by 1). The tumors having a mean area between 500 to 1000 are both benign and malignant, therefore show that the classification depends on more factors other than mean area. A linear regression line is also plotted for further analysis.
If we are going to do predictions based on the linear regression line drawn, we could set the threshold classifier output value at say, 0.5.
This means that tumors having a mean area less than the threshold( the area corresponding to 0.5 in the y-axis) will be classified as benign and those having a mean area greater than the threshold will be classified as malignant.
But from the plot, we can state the following reasons for linear regression not being a perfect model.
For values of x having a corresponding hypothesis value less than 0.5, predicting it as 0(or malignant)and also vice-versa is inappropriate.We can also see that the hypothesis value is greater than 1 and less than 0 in some cases which can't be true(as there are only two classes 0 and 1).
For values of x having a corresponding hypothesis value less than 0.5, predicting it as 0(or malignant)and also vice-versa is inappropriate.
We can also see that the hypothesis value is greater than 1 and less than 0 in some cases which can't be true(as there are only two classes 0 and 1).
This is where logistic regression comes into the picture. Logistic regression is a regression model specifically used for classification problems i.e., where the output values are discrete.
We observed form the above part that, while using linear regression, the hypothesis value was not in the range of [0,1]. This is would be the basic requirement of Logistic regression.
This means that all the predictions are supposed to be between 0 and 1.
The hypothesis of linear regression is given by:
For logistic regression, the above hypothesis is modified a little:
where:
z is a real number. g(z) is called the Sigmoid function or Logistic function(which gives us the name Logistic regression).
Putting together both the equations given above:
Let’s see the shape of the logistic function:
def sigmoid(x): return 1/(1+ np.exp(-x))x = np.linspace(-10,10,num = 1000)fig = plt.figure(figsize = (10,10))sns.set(style = 'whitegrid')sns.lineplot(x = x, y = sigmoid(x))
We can infer the following from the graph:
It crosses the y-axis at 0.5.The logistic function asymptotes at 1 as z tends to infinity and at 0 as z tends to negative infinity.As g(z) take on values in the range of (0,1), the values of h(x) also lies between (0,1).
It crosses the y-axis at 0.5.
The logistic function asymptotes at 1 as z tends to infinity and at 0 as z tends to negative infinity.
As g(z) take on values in the range of (0,1), the values of h(x) also lies between (0,1).
So given the hypothesis:
we need to find the parameters which fit the training examples so that the hypothesis can be used to make predictions.
The output given by the hypothesis h(x) can be interpreted as the probability for y = 1 for a given input x.
Consider the breast cancer data set. The feature vector for the feature mean area can be formed as:
Assume the hypothesis h(x) gives us a value of 0.7 for the above feature vector. As we have denoted malignant with 1 and benign with 0 we can say that the probability of the tumor being malignant is 70%.
This can be mathematically represented as follows:
The RHS can be read as
probability that y=1, given x, parameterized by theta
Since this a classification problem, we know that y can be equal to only two values 0 or 1(in this case). This applies to both the training examples and also any future predictions that we make. So given the hypothesis h(x), we can compute the probability for y=0 also as follows:
Consider the logistic function that we plotted above:
Let’s understand better when the hypothesis makes predictions that y=1 and y=0.
Suppose the prediction y=1 happens when
From the plot, we notice that the above condition occurs as:
i.e., when z is positive, g(z) take on values greater than 0.5.
So considering our hypothesis h(x), we can say that:
Similarly, assume prediction y=0 happens when
By similar argument as above:
We can use the above conclusions to better understand how the hypothesis of logistic regression makes predictions.
Consider the simple training set shown below:
os = np.array([[0, 0.5], [0.5, 0.5], [1.5, 0.5], [1, 0.5], [0.5, 0.5], [0.5, 1.5], [0, 1.5], [0,2.5]])xs = np.array([[1.5, 3], [2, 3.5], [2, 3], [2, 2.5], [2.5, 3.5], [3, 3], [3, 1.5], [3, 2], [3, 1], [3.5, 1.5]])fig = plt.figure(figsize = (10,10))sns.set(style = 'whitegrid')ax = sns.scatterplot(x = os[:,0], y = os[:,1], marker = 's', s = 100, color = 'r')ax = sns.scatterplot(x = xs[:,0], y = xs[:,1], marker = 'x', s = 100, color = 'k')ax.set(xlabel = 'x1', ylabel = 'x2')
We will assume the X’s correspond to y = 1 and the squares correspond to y = 0. Consider its hypothesis to be:
where x1 and x2 are the two features.
Assume we end up choosing the parameters that fit the equation to be(the process of choosing the parameters will be discussed later):
The parameter vector is going to be:
Taking reference from the arguments in the above section, prediction y=1 happens when:
From the parameters that we ended up with, we get
Plotting the line x1+x2 = 3:
x = np.linspace(0, 3, num = 10)ax = sns.lineplot(x = x, y = 3-x)
We can see that the region to the right of the line corresponds to the condition x1+x2≥3 and the hypothesis would predict y=1 if it were in this region and vice-versa. This line that divides both the regions is called the decision boundary of this training dataset and it corresponds to h(x)=0.5 exactly.
To be very clear, the decision boundary and the region are the properties of the hypothesis and its parameters and not that of the training dataset.
Consider a somewhat complex data-set given below:
os = np.array([[0,0], [0,0.5], [0.5,0.5], [0.5,0], [-0.5,0.5], [-0.5,0], [-0.5,-0.5], [0,-0.5], [0.5, -0.5],])xs = np.array([[1,1], [-1,1], [1,-1], [-1,-1], [0,1.5], [-1.5,0], [0,-1.5], [1.5,0]])fig = plt.figure(figsize = (10,10))sns.set(style = ‘whitegrid’)ax = sns.scatterplot(os[:,0], os[:,1], marker = ‘s’, s = 100, color = ‘r’)ax = sns.scatterplot(xs[:,0], xs[:,1], marker = ‘x’, s = 100, color = ‘k’)
As in the above example, the X’s belong to the region y=1 and the squares belong to the region y=0. Let’s say our hypothesis looks like:
Assume we end up choosing the values of parameters as:
Then our parameter vector looks like:
As per our earlier discussion, the hypothesis will predict y=1 when:
We can see that the above equation corresponds to a circle of radius 1 centered at the origin.
Plotting the equation:
theta = np.linspace(0, 2*np.pi , 1000)r = np.sqrt(1)x = r*np.cos(theta)y = r*np.sin(theta)ax.plot(x,y, color = 'b')
The circle is our decision boundary and the region outside the circle corresponds to y=1 and inside the circle corresponds to y=0. We can see that the decision boundary doesn't necessarily have to be a straight line but also more complex shapes like a circle, ellipse, and any other irregular shapes. Again the decision boundary is a property of the hypothesis and its parameters and not that of the training dataset.
The cost of a logistic regression problem is given by:
where, as discussed before h(x) is the prediction of the hypothesis and y is the actual class label.
Let’s plot this function to see how it corresponds to each case. We need to set the limits to h(x) as [0,1] as it lies in that range for logistic regression.
If y=1
If y=1
x = np.linspace(0,1, num = 100)fig = plt.figure(figsize = (10,10))sns.set(style = 'whitegrid')ax = sns.lineplot(x = x, y = -np.log(x))ax.set(xlabel = 'h(x)', ylabel = 'cost')
The following inferences can be made from the plot:
a. Cost becomes zero as h(x)=1 and y=1. This is obvious since the hypothesis predicts y as 1 which is true, then the cost will be zero.
b. As h(x) approaches zero the cost tends to infinity. This happens as this plot is specific to y=1 but, when h(x) predicts it as 0 the cost tends to infinity.
2. If y=0
ax = sns.lineplot(x = x, y = -np.log(1-x))
As discussed above, the following can be inferred:
a. Cost becomes zero as h(x)=0 and y=0.
b. Cost tends to infinity as h(x) approaches 1 since y=0.
The cost function of logistic regression is given by:
The point to note is that y=0 or 1 always in a classification problem having 2 classes.
The cost can be represented in a single line as follows:
This is a more compact representation of the cost. Thereby the cost function can be written as follows:
We need to find the parameters so that the value of cost function is minimized. This can be mathematically represented as:
The differentiation term is given by:
Plugging it in the gradient descent equation:
Let’s use the above equations to execute Logistic Regression on the breast_cancer dataset.
x = df.area_meany = df.diagnosisx = preprocessing.scale(x)theta_0_gd = 0theta_1_gd = 0alpha = 0.01h_theta_0_gd = 1h_theta_1_gd = 1epoch = 0m = len(x)fig = plt.figure(figsize = (10,10))sns.set(style = 'whitegrid')ax = sns.scatterplot(x,y)while h_theta_0_gd != 0 or h_theta_0_gd != 0:    if epoch > 5000:        break    h_theta_0_gd = 0    h_theta_1_gd = 0    for i in range(len(x)):        h_theta_0_gd += ((1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x[i]))))) - y[i])        h_theta_1_gd += (((1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x[i]))))) - y[i]) * x[i])    h_theta_0_gd = (1/m) * h_theta_0_gd    h_theta_1_gd = (1/m) * h_theta_1_gd    theta_0_gd -= (alpha * h_theta_0_gd)    theta_1_gd -= (alpha * h_theta_1_gd)    epoch += 1ax = sns.lineplot(x,(1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x))))) , color = 'r', linewidth = 3)
The values of theta_0 and theta_1 are :
print(theta_0_gd, theta_1_gd)output>>-0.4173702618170074 3.0623106036104937
We could also plot the value of cost function after each epoch to check the convergence of the algorithm:
x = np.array(df.area_mean)y = np.array(df.diagnosis)x = preprocessing.scale(x)theta_0_gd = 0theta_1_gd = 0alpha = 0.01h_theta_0_gd = 1h_theta_1_gd = 1epoch = 0Epoch = []m = len(x)j = 0J = []fig = plt.figure(figsize = (20,10))ax1 = fig.add_subplot(1,2,1)ax2 = fig.add_subplot(1,2,2)style.use('ggplot')ax1.scatter(x,y)while h_theta_0_gd != 0 or h_theta_0_gd != 0:    if epoch > 5000:        break    h_theta_0_gd = 0    h_theta_1_gd = 0    for i in range(len(x)):        h_theta_0_gd += ((1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x[i]))))) - y[i])        h_theta_1_gd += (((1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x[i]))))) - y[i]) * x[i])    h_theta_0_gd = (1/m) * h_theta_0_gd    h_theta_1_gd = (1/m) * h_theta_1_gd    theta_0_gd -= (alpha * h_theta_0_gd)    theta_1_gd -= (alpha * h_theta_1_gd)    for i in range(m):        j += ((y[i] * (np.log(1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x[i]))))))) - ((1-y[i]) * np.log(1-(1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x[i]))))))))    J.append((-1/m) * j)    epoch += 1    Epoch.append(epoch)ax1.scatter(x,(1/(1+np.exp(-(theta_0_gd + (theta_1_gd * x))))) , color = 'k')ax2.plot(Epoch,J)plt.show()
We can see that the algorithm converged at around 500 epochs where the cost function reached a minimum value of -30, after which it started increasing till 5000 epochs. This happened as the learning rate ( which is 0.01) is very large so the algorithm after a certain point starts diverging. Consider lowering the learning rate further.
The above plot corresponds to a learning rate of 0.001. We can see that the algorithm converges at around 6000 epochs at a cost function value of -375, after which it starts increasing again. We can conclude from this that decreasing the learning rate helps in fine-tuning the parameters to obtain the more minimized value of cost function, but the algorithm takes a larger no of epochs i.e., takes more time to converge.
Let’s check if the parameter values obtained in our model are on par with the sklearn model of Logistic regression
X = x.reshape(-1,1)x_train, x_test, y_train, y_test = model_selection.train_test_split(X, y, test_size = 0.33)clf = LogisticRegression(max_iter = 5000)clf.fit(x_train, y_train)clf.coef_output>> array([[3.5028]])clf.intercept_output>> array([-0.31180412])
As we can see, the parameters are having close enough values when compared. We could even plot the regression line using the parameters obtained to check if we are getting a similar plot.
Theta_1 = clf.coef_Theta_0 = clf.intercept_fig = plt.figure(figsize = (10,10))ax = sns.scatterplot(X,y)ax = sns.lineplot(X, (1/(1+np.exp(-(Theta_0 + (Theta_1[0] * X))))), color = 'r')
In this article, we have seen what is meant mathematically by a Classification problem, how linear regression is of little use in the case of a classification problem, Logistic regression and its hypothesis, cost, cost function, decision boundary, and gradient descent. We also built a Logistic regression model form scratch and trained it using the Breast cancer dataset. We also used the inbuilt model from sklearn for verification.
I am learning most of the concepts about Machine learning from this Youtube playlist. It is quite helpful and easy to understand too.
I am learning most of the concepts about Machine learning from this Youtube playlist. It is quite helpful and easy to understand too.
scikit-learn.org
2. Consider checking out the official docs of sklearn’s Logistic regression classified for further usability.