Logistic regression
By: Joanna Trojak
Tags: Machine Learning Ai, Logistic Regression, Machine Learning Python, Artificial Intelligence, Machine Learning

Logistic regression is a well-known classification algorithm. It predicts the probability that a certain instance belongs to a particular class, for instance, what is the probability that the email is spam. Such classifier is called the binary classifier.

Estimating Probabilities

Similarly to linear regression, logistic regression computes a weighted sum of the input features but instead outputting the results directly, it outputs the logistic of the result. What does it mean? The logistic is a sigmoid function that outputs the number between 0 and 1. Once the model has estimated the probability of the particular instance belonging to the positive class it can make the prediction easily.

Training and Cost Function

The objective of the training is to set the parameter vector θ so that the model estimates the high probabilities for positive examples and low probabilities for negative examples.

This equation makes sense because -log(t) will be very large when t approaches 0. In other words, the cost will be large if the model approximates the probability close to 0 (for a positive example) and it will also be very large if the model estimates the probability close 1 (for negative instances). On the other hand, — log(t) is close to 0 when t is close to 1, so the cost will be close to 0 if the estimated probability is close to 0 for a negative instance or close to 1 for a positive instance, which is precisely what we want.

The cost over the whole training set is just the average cost over all the training instances. There is no normal equation but the good news is that the function is convex so the gradient descent should be able to find the global minimum.

Decision Boundaries

We can illustrate the logistic regression with iris dataset. The dataset contains the sepal and petal length and width of 150 iris flowers of three different species: Iris-Setosa, Iris-Versicolor, and Iris-Virginica. We can build a classifier that detects Iris-Virginica based on the petal width feature.

First, we have to load the dataset:

Now, we have to build the logistic regression model.

After that, we are going to look at the model estimated probabilities. The petal of Iris-Virginica is between the 1.4 cm to 2.5 cm, while the other types of the iris are between the 0.1 cm to 1.8 cm.

We can see that there is a slight overlap. Above 2cm the classifier is highly confident that the flower is Iris-Virginica while below 1cm it is convinced that it is not. In between the classifier is unsure but when we call the predict() method it will output the probability of belonging to the certain class. 1.6 cm is the mark for the decision boundary where both types of flowers are equally probable.

The following diagram displays the decision boundary diagram for two features petal width and length.

The dashed line represents the decision boundary where there is a 50% probability of particular example belonging to either of class.

Softmax Regression

So far we have seen the binary classifier with logistic regression. Softmax regression, on the other hand, can support the multiclass classifier without training multiple binary classifiers.

First of all, the softmax regression model computes the score sk(x) for each class k, then it estimates the probability of each class by applying the softmax function to the scores.

We should keep in mind that the softmax regression computes one class at the time so it should be used for mutual exclusive classes such as different types of flowers. We cannot recognize multiple people in one picture.