Understanding Logistic Regression
By: Dorian Lazar
Tags: Logistic Regression, Artificial Intelligence, Mathematics, Data Science, Machine Learning

What is logistic regression? Logistic regression is just adapting linear regression to a special case where you can have only 2 outputs: 0 or 1. And this thing is most commonly applied to classification problems where 0 and 1 represent two different classes and we want to distinguish between them.

Linear regression outputs a real number that ranges from -∞ to +∞. And we can use just this even in the 0/1 classification problem: if we get a value >= 0.5 report it as class label 1, if the output is < 0.5 report it as a 0.

Where x is a vector of features (plus a component with constant 1 for bias) of one observation, and w is the weights vector.

But, we can get slightly better results both in terms of accuracy and interpretability if we squash the regression line into an “S”-shaped curve between 0 and 1. We squash the regression line by applying the sigmoid function to the output value of a linear regression model.

Below is the sigmoid function:

More exactly, we compute the output as follows: take the weighted sum of the inputs, and then pass this resulting number into the sigmoid function and report the sigmoid’s output as the output of our logistic regression model.

This procedure helps us both in getting slightly better accuracy and in the interpretability of the output. If our model outputs any real number like -5 or 7, what these numbers actually mean? What can we tell about our two classes: 0 and 1?

But, when we have outputs between 0 and 1, we can interpret them as probabilities. The output of a logistic regression model is the probability of our input belonging to the class labeled with 1. And the complement of our model’s output is the probability of our input belonging to the class labeled with 0.

Where y is the true class label of the input x.

OK. So, by now we have seen how a logistic regression model obtains its outputs, given the input. But what about its weights? What weights should it have to make good predictions?

Our model needs to learn those weights, and the way it learns is by giving to our model an objective function, then it finds the weights that minimize or maximize this objective.

There are many ways we can come up with an objective function, especially if we consider adding regularization terms to our objective.

In this article, we’ll explore only 2 such objective functions.

First, let’s write our logistic regression model as follows:

Where X is a matrix that contains all our observations as rows, and columns represent the features. y hat is the output of our model, it is a vector that contains the predictions for each observation.

Let’s rewrite our logistic regression equation in the following way:

The operations on the right-hand side of the last line are element-wise.

What do you observe in the last line above? If we apply the function on the right-hand side of the last equation on the labels for logistic regression and consider the output of this function application as the new labels, then we obtain a linear regression. So, we can use the sum of squared errors as a loss function and find the weights that minimize it. We can find the weights by using either a closed-form formula or SGD (stochastic gradient descent) as you can read more about in the following article on linear regression:

Below are the closed-form solution and the gradient of the loss (that we can use in the SGD algorithm) for linear regression:

For logistic regression we just need to replace the y in these 2 equations above with the right-hand side of the previous equation:

When we apply these formulas, we provide the true labels for y hat.