Animations of Logistic Regression with Python
By: Tobias Roeschl
Tags: Machine Learning, Regression, Statistics, Python, Animation

This article is about creating animated plots of simple and multiple logistic regression with batch gradient descent in Python. In the end, I will also present a visual explanation of why the cross-entropy cost function is the method of choice to quantify costs in logistic regression. In terms of structure and content, this article relates to — and partially builds on — previous articles I wrote about creating animations of batch gradient descent with the example of simple linear and multiple linear regression. The general idea is to set up a logistic regression model and train the model on some arbitrary training data while storing parameter values and costs for each epoch. After confirming our results through sklearn’s built-in logistic regression model, we will use the stored parameter values to generate animated plots with Python’s celluloid module. The animations we use to visualize logistic regression will be similar to the ones we created for previous articles on linear regression.

Setting up the model:

Logistic regression is a classification algorithm that predicts probabilities of particular outcomes given one or more independent variables. The independent variable can be continuous or categorical. The outcome can be interpreted as taking membership in one of a discrete set of classes. In this article, we confine the number of classes to two classes although it is theoretically possible to generalize logistic regression to multiclass problems with more than two possible outcomes. Probabilities can be calculated with the sigmoid function, which is a special case of the logistic function. For a given, single measurement X and n independent predictor variables, the probability of the response variable Y equaling a case (“1”) rather than a non-case (“0”) can be expressed as:

with our predictor variable X and our weights w given as:

and a scalar b representing the bias term (y-intercept). As with linear regression, we try to find the optimal model parameters θ, consisting of our weight(s) w and our bias b, to minimize the costs J of our model. This can be achieved by various optimization algorithms. One of these algorithms is batch gradient descent, where we adjust parameter values proportional to the negative gradient of our cost function until we reach model convergence. Mathematically, this can be expressed with the following formula:

with ∇J(θ) representing the gradient of our cost function and α representing the learning rate. In this equation, e is representing the respective epoch. So far, so good. Logistic regression, however, differs decisively how we define our cost function. While we technically could apply the mean squared error (MSE)-method, like we did with linear regression, this would result in a non-convex cost function in the case of logistic regression. Non-convex cost functions may have multiple local minima. Accordingly, gradient descent is not guaranteed to converge to the global minimum with non-convex cost functions.¹ In order to overcome this issue, the concept of cross-entropy was introduced to quantify costs in logistic regression. For a dichotomous outcome variable y, the costs J can be calculated as follows:

with pᵢ∈[ 0;1] representing the model’s prediction for each of our N samples, on which the model is trained. Since we aim to compute the gradient of our cost function, we need to determine the partial derivatives of J with respect to a particular weight wⱼ and b and get the following:

In the following PDF, I gave a detailed explanation of how to derive the partial derivatives of the binary cross-entropy loss functions with respect to its parameters.

In Python, we can import some libraries and define our model. In order to set up a logistic regression model, which is flexible to the number of independent variables, we introduce a weight matrix w with one weight wⱼ for every input variable. I decided to arbitrarily set the initial parameter values for the weights to 0 and 0.5 for the bias:

Simple logistic regression

Equivalently to simple linear regression, we have one predictor variable in simple logistic regression. In Python, we introduce our training data and fit our model to the data:

The learning rate is intentionally set to a particularly small value of α=0.0001 in order to avoid large steps especially at the beginning of our animations. In order to compare the results of our simple logistic regression model to those we get with sklearn’s model, we display both models’ results after their respective fitting process.

Since the results of both models are consistent with each other, we can begin to create our first animation:

Logistic regression curve and surface plot of costs (large)

In the upper half of the animation, we can observe how the logistic regression curve is fitted to the training data. By defining which epochs are being used for the animations, we can smoothen the temporal sequence of the fitting process which results in more appealing animations. I thought it was useful to draw dashed connection lines between actual data points and those predicted by the model. It is worth mentioning that in logistic regression our goal is not to minimize the (squared) distances represented by these connection lines since we are using a completely different cost function. I would like to come back to this in more detail later on. In the lower half of the animation, we can see how costs drop simultaneously after each epoch and finally end up in the global minimum of the surface plot. The surface plot portrays the costs for a given range of respective parameter values given our training data. In the literature, these surface plots are also being referred to as loss landscapes. In Python, we can create loss landscapes by calculating the costs for particular combinations of two model parameters via meshgrids. In the case of simple logistic regression, the model parameters are the weight and the bias term.

Multiple logistic regression

Multiple logistic regression analysis applies when there is more than one predictor variable. In the following example, we will fit our model to a training dataset with two independent variables. Since we can only portray costs for two parameters at once in our three-dimensional animations, we have to keep one parameter fixed. Therefore, we define yet another model — this time with a fixed intercept— and also train this new model on the new training data. In the new model, the part of code where b is being updated is removed. The bias term is set to the bias the former multiple logistic regression model converged to. Theoretically however, we could use any other value for the fixed bias term. The parameter values we obtain during the fitting process are once again stored in arrays.

Like we did before, we return the final model parameters and costs and compare it to the results we get with sklearn’s model:

In multiple logistic regression, we intend to fit a 3D curve to our training data. For this reason, we need to calculate y-values for yet another meshgrid, this time spanned by x₀- and x₁-values. Lastly, we can give out the final parameter values and costs portrayed in the animations to ensure that we approximately visualized model convergence despite substantially restricting the number of epochs used to create the animations (see commented-out code!).

Logistic regression curve in 3D and surface plot of costs (large)

Additionally, we can also portray the path of gradient descent via a contour plot.

Logistic regression curve in 3D and contour plot of costs (large)

Cross-entropy cost function (CE) vs. Mean squared Error (MSE)

Finally, let me come back to why we need the cross-entropy (CE) loss function in logistic regression. First and foremost, it can be shown mathematically that the CE-cost function is convex with exactly one minimum, which is the global minimum. In contrast, applying the MSE-cost function on logistic regression results in a non-convex cost function.² In the following, I will use a graphical approach to compare both methods on the example of our training data.

In Python, we introduce a new cost-function “MSE_cost( )” to quantify costs by the use of MSE. Like we did before with the cross entropy-cost function we can then create loss landscapes with respect to our training data (x_train2, y_train2) and our weights (w₀,w₁). I intentionally increased the range of possible values for our weights regarding the MSE-method to w₀,w₁ ∈ [-5,5] since this will help to illustrate the differences of both loss landscapes. Furthermore, numerical instability is not much of a concern with MSE as it is with CE.³

Loss landscapes created with CE-cost function (left) and MSE-cost function (right)

Obviously, the loss landscape on the right is looking “bumpier” now compared to the smooth and convex loss landscape of the cross-entropy cost function. For a more detailed view, we can try to visualize the MSE-loss landscape with a contour plot:

Contour plot of MSE-cost function with ‘global’ minimum (“x”) and local minimum (*).

First, we can recognize that there is a minimum (“x”), located at roughly the same point where the CE-loss landscape had its global minimum (0.089,0.19). I will refer to this point, marked by “x” as the ‘global’ minimum of the MSE-loss landscape. I put ‘global’ in parenthesis because we lack the mathematical proof that this point actually is the global minimum of the MSE-loss landscape. Without further investigation, we could also assume that there is a local minimum in close proximity to the asterisk. With starting values for our weights within this area, gradient descent might get stuck and not converge to the ‘global’ minimum (“x”) in the middle of the contour plot. The following animation once more illustrates the full extent of non-convexity of the MSE-cost function above:

Loss landscape calculated with MSE-cost function (code)

I will address gradient descent with the example of non-convex cost functions in more detail in my next article about neural networks.

I hope you found this article helpful. Should any questions arise or if you noticed any mistakes, feel free to leave a comment. The complete notebook can be found on my GitHub. Thank you for reading!

References:

Appendix: