Using Logistic Regression to Create a Binary and Multiclass Classifier from Basics | by Naveen Venkatesan | Towards Data Science

As data science and machine learning have become an integral part of many fields in industry and academic research, basic literacy in these techniques can be very fruitful to identify trends in data, especially when the size of datasets rapidly increase. Having a background in experimental science, linear regression always seemed fairly intuitive, as we often fit our experimental data to theoretical models to extract material properties. Making a classifier (i.e. predicting if Team A will beat Team B based on several prior metrics), however, was not something I had used or had much experience with until I did some reading. In this article, I hope to succinctly describe how to create a classifier from scratch using logistic regression. In practice, you will probably use a package to do this such as scikit-learn or tensorflow, but understanding some of the underlying equations and algorithms will be extremely helpful in knowing what is going on “under the hood”.
Much of the work in this article was inspired by lessons from the Machine Learning class on Coursera taught by Andrew Ng.
We will use Python for all the code in this article, so let’s import all the packages that we will need:
Making discrete predictions is something we do all the time, probably without even thinking too much about how. For example, you’re looking at the menu at a restaurant — you read descriptions of different menu items, remember some tips from a food review you recently read, maybe ask the waiter a few questions, and make a discrete decision of what you want to order. If you have never eaten at this restaurant, you are essentially making an informed prediction based on what you think you will like.
To achieve similar behavior in our predictor algorithm, a great place to start would be to pick a mathematical function that essentially gives a binary output or 0 or 1. To do this, we can use the logistic or sigmoid function, which has the form:
We can code this function in Python and plot it as follows:
As shown in the plot, the sigmoid function rapidly changes from an output of near 0 to near 1 around x = 0. Since the output values are symmetric around y = 0.5, we can use this as the threshold for making out decision, where y ≥ 0.5 outputs 1 and y < 0.5 outputs 0.
To see this in action, let’s train some data!
The scenario we have is as follows — we are building a simple movie recommendation system that takes into account an average user score between 0 and 5 (across all users) and an average critic score between 0 and 5. Our model should then generate a decision boundary based on our input data to predict whether the current user will like the movie, and recommend it to them.
We will a set of random user and critic scores for 100 movies:
Now, let’s generate classifications — in this case, either the user liked the movie (1), or did not (0). To do so, we will use the following decision boundary function. We set the right hand side of this equation to 0 because it defines when the logistic function is equal to 0.5:
Note that in a real dataset, you would have no idea what the decision boundary’s functional form is. For the sake of this tutorial, we are defining a function and adding some noise to it so that the classification is not “perfect”.
Now, we can plot our initial data, where an orange circle represents a movie liked by the user, and a blue circle a movie that was not liked:
To determine how good our decision boundary is, we need to define a penalty for wrong predictions, which we will do by creating a cost function. Our predictions will be made as follows:
When P ≥ 0.5, we output 1 and when P < 0.5, we will output 0, where w0, w1, and w2 are the weights we are optimizing for.
To penalize wrong classifications, we can take advantage of the logarithm function, since log(1) = 0 and log(0) → -∞. We can use this to create two penalty functions as follows:
We can visualize this to more clearly see the effects of these penalty functions:
Using the facts that our outputs (y) are going to be either 0 or 1, we can elegantly combine these two penalty functions in one expression that encompasses both cases:
Now, when our output is 1 and we predict something close to 0 (first term), we incur a steep penalty — similarly, the same scenario occurs when our output is 0 and we predict something close to 1 (second term).
We can now take our penalty function and generalize it to m training examples — the label of the i-th training example is indicated by (i). We will also divide the total cost by m to get the mean penalty (just like in mean squared error in linear regression). This final expression is also known as the cost function. We will refer to our two features (user score and critic score) as x1 and x2.
To find the optimal decision boundary, we must minimize this cost function, which we can do with an algorithm called gradient descent, which essentially does two things: (1) finds the direction of maximum decrease by calculating the gradient of the cost function, and (2) update the weight values by moving along this gradient. To update the weights, we also provide a learning rate (α), which determines how much we move along this gradient. There is a trade-off in choosing a learning rate — too small and our algorithm takes too long to converge; too big, and our algorithm can actually diverge. More information about both gradient descent and the learning rate can be found at the linked articles. We therefore have the following update rule for each weight:
The final piece of the puzzle is calculating the partial derivatives in the above expressions. We can use the chain rule from calculus to break this down as follows:
Now, we can put it all together to get the following gradient expression and gradient descent update rule:
We can now initialize our variables to minimize the cost function.
All the gradient descent operations above are conducive to using matrix multiplication and linear algebra. First, we initialize our variables as following:
We can now rewrite the two earlier expressions for the cost function, gradient, and update function for the weights as follows, where the superscripted T signifies the transpose of the matrix:
Our variables can be initialized as follows (we will set all the weights equal to 0 to begin):
Now that the long wall of math is out of the way, let’s train our classifier model!
We must first define functions for the cost function and the gradient descent update. The following function calculates the cost and the gradient values and returns both:
When defining the gradient descent function, we must add an additional input called num_iters which tells the algorithms the number of iterations to take before returning the final optimized weights.
Now, after doing all that hard work, the following line trains our model!
J, w_train = gradient_descent(X, y, w, 0.5, 2000)
In a real situation, we would have split our training data in order to cross-validate and test our model, but for the purposes of this tutorial, we will use all of the training data.
To ensure that we are in fact decreasing the value of the cost function with each iteration, we can plot this:
And now the moment of truth — we can plot our decision boundary. The way that we will overlay this on our original plot is by defining a grid of points and then calculating the prediction value from the sigmoid function using our trained weights. We can then plot the contour where this prediction value is equal to 0.5.
Et, voilà!
We can now write a function to make predictions based on this trained model:
Now, let’s make a few predictions on new examples:
X_new = np.asarray([[1, 3.4, 4.1], [1, 2.5, 1.7], [1, 4.8, 2.3]])print(predict(X_new, w_train))>>> [[1.]       [0.]       [1.]]
Now that we’ve done all the hard work of making a binary classifier, extending this to more classes is fairly straightforward. We will use a strategy called one-vs.-all classification, where we train a binary classifier for each distinct class and choose the class that has the largest value returned by the sigmoid function.
Let’s build on the model from our binary example — this time the user can give a movie a score of 0-stars, 1-star, or 2-stars and we are trying to determine the decision boundaries based on user and critic scores.
First, we generate data again and apply two decision boundaries:
Our dataset looks as follows, where blue circles represent 0-stars, orange circles represent 1-star, and red circles represent 2-stars:
For each binary classifier that we train, we will need to relabel the data such that the outputs for our class of interest is set to 1 and all other labels are set to 0. As an example, if we have 3 groups A (0), B (1), and C (2) — we must make three binary classifiers:
(1) A set to 1, B and C set to 0
(2) B set to 1, A and C set to 0
(3) C set to 1, A and B set to 0
We have a function to relabel our data for each classifier:
Now, we do the same model training as in the binary classification section earlier, but three separate times:
Plotting the decision boundaries this time is a little more involved. We must calculate the prediction values at each point in our grid for each of our trained classifiers. We then choose the maximum prediction value at each point and assign it’s respective class based on which of the classifiers led to that maximum value. The contour lines we choose to plot are 0.5 (between 0 and 1) and 1.5 (between 1 and 2):
And after much anticipation, here are our trained decision boundaries!
Finally, let’s make a prediction function. To do this, we will use a little trick — we will make a prediction for each classifier in each of the examples. We will then append each set of predictions as a new column to a predictions matrix. Then, by going along each row and choosing the column index that has the maximum value, we automatically get out the classifier label!
X_new = np.array([[1, 1, 1], [1, 1, 4.2], [1, 4.5, 2.5]])print(predict_multi(X_new, [w_class1, w_class2, w_class3]))>>> [[0]       [2]       [1]]
There it is! A multiclass classifier made completely from scratch!
Thanks for reading! This article just scratches the surface of logistic regression and classification, but I hope that you enjoyed it. The examples presented can be found here. Again I owe a lot of the inspiration of this article to the Machine Learning class on Coursera taught by Andrew Ng.