Logistic Regression and Decision Boundary | by Anuradha Wickramarachchi | Towards Data Science

The fundamental application of logistic regression is to determine a decision boundary for a binary classification problem. Although the baseline is to identify a binary decision boundary, the approach can be very well applied for scenarios with multiple classification classes or multi-class classification.
In the above diagram, the dashed line can be identified as the decision boundary since we will observe instances of a different class on each side of the boundary. Our intention in logistic regression would be to decide on a proper fit to the decision boundary so that we will be able to predict which class a new feature set might correspond to. The interesting fact about logistic regression is the utilization of the sigmoid function as the target class estimator. Let us have a look at the intuition behind this decision.
The sigmoid function for parameter z can be represented as follows. Note that the function always lies in the range of 0 to 1, boundaries being asymptotic. This gives us a perfect output representation of probabilities too.
Now that we know our sigmoid function lies between 0 and 1 we can represent the class probabilities as follows.
Here θ stands for the estimated parameter vector and X is the vector of variables considered.
X = X0, X1 ... Xn    <- n Features and X0=1θ = θ0, θ1 ... θn    <- Parameters to be estimated
Now that we are aware of a function estimate for our probabilities we shall come up with a way to estimate the parameters represented by θ vector. For this exercise let us consider the following example.
We have a dataset with two features and two classes. Our objective is to discover the proper values of θ for the two features. This can be modelled as follows.
h(θ) is the probability estimation or the hypothesis function.
For a moment let’s assume that we can use the root mean squared error (RMS), similar to linear regression. You may refer to the following article for more insights.
towardsdatascience.com
For simplicity, I will plot the variation of cost function against θ[0] which is biased of our estimator. This is based on the representation of our target variable y to be as follows;
We can see that there are two local optima. This is unexpected and is caused by the behaviour of our sigmoid function. Therefore, the cost function is represented as follows which matches our expectations perfectly.
This is a piece-wise function that has different definitions at different values of y. The idea is to penalize the wrong classification exponentially. For example, for label y=1 if the model predicts h(x)=0, we will have the first equation reaching infinity and vice versa.
Now that we have a better loss function at hand, let us see how we can estimate the parameter vector θ for this dataset.
Since we know the loss function, we need to compute the derivative of the loss function in order to update our gradients. It can be done as follows. This whole operation becomes extremely simple given the nature of the derivate of the sigmoid function. It will leave us with the following loss function.
Note that this is the exact linear regression loss/cost function we discussed in the above article that I have cited. Since I have already implemented the algorithm, in this article let us use the python sklearn package’s logistic regressor.
The usage is pretty straightforward. However, it is important that we understand the estimated parameters. The model fitting can be done as follows. Here X is a 2-dimensional vector and y is a binary vector.
from sklearn.linear_model import LogisticRegressionclf = LogisticRegression(random_state=0).fit(X, y)
Estimated parameters can be determined as follows.
print(clf.coef_)print(clf.intercept_)>>> [[-3.36656909  0.12308678]]>>> [-0.13931403]
Coefficients are the multipliers of the features. We represented them in our θ vector in indices 1 and 2. Intercept is the bias value of the model.
Usage of the logistic regression after fitting can be done as follows.
clf.predict_proba([[ 0.8780991 ,  0.89551051]])>>> array([[0.95190421, 0.04809579]])
This is the prediction for each class. Note that the total probability is equal to one. The same can be achieved using the following implementation. h(theta, xi) is the hypothesis function using learned theta parameters.
def h(theta, xi):    return 1/(1 + np.exp(-1*np.dot(xi, theta)))
Note that I have used np.dot() to obtain the matrix or vector multiplication which is far more efficient than using a forloop. This is also called vectorization. We can obtain our p(y=1) estimate using the following function call.
h([-0.13931403, -3.36656909,  0.12308678], [1, 0.8780991 ,  0.89551051])
Note that I have used our intercept value as the first element of theta parameter and the rest in order. I have prepended an additional 1 for the feature vector which corresponds to the learned bias.
Finally, we can plot our boundary as follows.
Logistic regression can easily be extended to predict more than 2 classes. However, you will have to build k classifiers to predict each of the k many classes and train them using i vs other k-1 classes for each class.
Logistic regression is a fast machine learning techniqueMost of the implementations use faster optimizers apart from the simple gradient descent we discussedIt is always wise to check for the existence of a decision boundary. You might require a technique like PCA or t-SNE.
Logistic regression is a fast machine learning technique
Most of the implementations use faster optimizers apart from the simple gradient descent we discussed
It is always wise to check for the existence of a decision boundary. You might require a technique like PCA or t-SNE.
I hope you enjoyed reading this article on Logistic Regression. You can find the Jupyer notebook here.
Some interesting reading for the curious;