Animations of Logistic Regression with Python | by Tobias Roeschl | Towards Data Science

This article is about creating animated plots of simple and multiple logistic regression with batch gradient descent in Python. In the end, I will also present a visual explanation of why the cross-entropy cost function is the method of choice to quantify costs in logistic regression. In terms of structure and content, this article relates to — and partially builds on — previous articles I wrote about creating animations of batch gradient descent with the example of simple linear and multiple linear regression. The general idea is to set up a logistic regression model and train the model on some arbitrary training data while storing parameter values and costs for each epoch. After confirming our results through sklearn’s built-in logistic regression model, we will use the stored parameter values to generate animated plots with Python’s celluloid module. The animations we use to visualize logistic regression will be similar to the ones we created for previous articles on linear regression.
Logistic regression is a classification algorithm that predicts probabilities of particular outcomes given one or more independent variables. The independent variable can be continuous or categorical. The outcome can be interpreted as taking membership in one of a discrete set of classes. In this article, we confine the number of classes to two classes although it is theoretically possible to generalize logistic regression to multiclass problems with more than two possible outcomes. Probabilities can be calculated with the sigmoid function, which is a special case of the logistic function. For a given, single measurement X and n independent predictor variables, the probability of the response variable Y equaling a case (“1”) rather than a non-case (“0”) can be expressed as:
with our predictor variable X and our weights w given as:
and a scalar b representing the bias term (y-intercept). As with linear regression, we try to find the optimal model parameters θ, consisting of our weight(s) w and our bias b, to minimize the costs J of our model. This can be achieved by various optimization algorithms. One of these algorithms is batch gradient descent, where we adjust parameter values proportional to the negative gradient of our cost function until we reach model convergence. Mathematically, this can be expressed with the following formula:
with ∇J(θ) representing the gradient of our cost function and α representing the learning rate. In this equation, e is representing the respective epoch. So far, so good. Logistic regression, however, differs decisively how we define our cost function. While we technically could apply the mean squared error (MSE)-method, like we did with linear regression, this would result in a non-convex cost function in the case of logistic regression. Non-convex cost functions may have multiple local minima. Accordingly, gradient descent is not guaranteed to converge to the global minimum with non-convex cost functions.1 In order to overcome this issue, the concept of cross-entropy was introduced to quantify costs in logistic regression. For a dichotomous outcome variable y, the costs J can be calculated as follows:
with pi∈[ 0;1] representing the model’s prediction for each of our N samples, on which the model is trained. Since we aim to compute the gradient of our cost function, we need to determine the partial derivatives of J with respect to a particular weight wj and b and get the following:
In the following PDF, I gave a detailed explanation of how to derive the partial derivatives of the binary cross-entropy loss functions with respect to its parameters.
Activate your 30 day free trial to continue reading.


Facebook




Twitter




LinkedIn


Share
Email


 



 





What to Upload to SlideShare
by SlideShare
13906958 views







Be A Great Product Leader (Amplify,...
by Adam Nash
2029108 views







Trillion Dollar Coach Book (Bill Ca...
by Eric Schmidt
1901745 views







APIdays Paris 2019 - Innovation @ s...
by apidays
2587128 views







A few thoughts on work life-balance
by Wim Vanderbauwhede
1792952 views







Is vc still a thing   final
by Mark Suster
1606573 views













In Python, we can import some libraries and define our model. In order to set up a logistic regression model, which is flexible to the number of independent variables, we introduce a weight matrix w with one weight wj for every input variable. I decided to arbitrarily set the initial parameter values for the weights to 0 and 0.5 for the bias:
Equivalently to simple linear regression, we have one predictor variable in simple logistic regression. In Python, we introduce our training data and fit our model to the data:
# Introduce training data:  
x_train = np.array([
    [-80],
    [-70],
    [-50],
    [-39],
    [-27],
    [-15],
    [-9],
    [12],
    [25],
    [36],
    [52],
    [65],
    [78],
    [90],
    [99],
    [110]
])

y_train = np.array([
    [0],
    [0],
    [0],
    [0],
    [0],
    [0],
    [1],
    [0],
    [1],
    [0],
    [1],
    [0],
    [1],
    [1],
    [1],
    [1]
])

xs=np.array([np.linspace(-150,200)]) # x-values later used for regression curve plot

# Fit model to training data: 
model=LogisticRegression(x_train,y_train, lr=0.0001) # set up model and define learning rate
model.fit(x_train,y_train, numberOfEpochs=700000) # set number of epochs

# Store parameter values in new variables:    
w=model.AllWeights.T
b= model.AllBiases
c=model.AllCosts
cl=model.All_cl

# Print results: 
print("Final weight: "+ str(np.float(model.w))) 
print("Final bias: "+ str(model.b))
print("Final costs: " + str(model.cost(x_train,y_train)))

Final weight: 0.0358378946387266
Final bias: -1.116903840009302
Final costs: 0.40058199922669563

The learning rate is intentionally set to a particularly small value of α=0.0001 in order to avoid large steps especially at the beginning of our animations. In order to compare the results of our simple logistic regression model to those we get with sklearn’s model, we display both models’ results after their respective fitting process.
# cross-check results with sklearn's inbuilt logistic regression model: 
from sklearn.linear_model import LogisticRegression
# - set C (= Inverse of regularization strength) to a very high number
# - use np.ravel() to prevent DataConversionWarning
clf = LogisticRegression(solver="lbfgs", random_state=0, C = 1e20).fit(x_train, y_train.ravel())
print(clf.coef_, clf.intercept_)

pred=expit(x_train @ clf.coef_.T + clf.intercept_) # calculate respective costs ...
#... for sklearn's fitted model parameters
print(- np.mean(y_train*np.log(pred) + (1-y_train)*np.log(1-pred)))

[[0.03586354]] [-1.11926705]
0.40058175764105697

Since the results of both models are consistent with each other, we can begin to create our first animation:
In the upper half of the animation, we can observe how the logistic regression curve is fitted to the training data. By defining which epochs are being used for the animations, we can smoothen the temporal sequence of the fitting process which results in more appealing animations. I thought it was useful to draw dashed connection lines between actual data points and those predicted by the model. It is worth mentioning that in logistic regression our goal is not to minimize the (squared) distances represented by these connection lines since we are using a completely different cost function. I would like to come back to this in more detail later on. In the lower half of the animation, we can see how costs drop simultaneously after each epoch and finally end up in the global minimum of the surface plot. The surface plot portrays the costs for a given range of respective parameter values given our training data. In the literature, these surface plots are also being referred to as loss landscapes. In Python, we can create loss landscapes by calculating the costs for particular combinations of two model parameters via meshgrids. In the case of simple logistic regression, the model parameters are the weight and the bias term.
Multiple logistic regression analysis applies when there is more than one predictor variable. In the following example, we will fit our model to a training dataset with two independent variables. Since we can only portray costs for two parameters at once in our three-dimensional animations, we have to keep one parameter fixed. Therefore, we define yet another model — this time with a fixed intercept— and also train this new model on the new training data. In the new model, the part of code where b is being updated is removed. The bias term is set to the bias the former multiple logistic regression model converged to. Theoretically however, we could use any other value for the fixed bias term. The parameter values we obtain during the fitting process are once again stored in arrays.
Like we did before, we return the final model parameters and costs and compare it to the results we get with sklearn’s model:
In multiple logistic regression, we intend to fit a 3D curve to our training data. For this reason, we need to calculate y-values for yet another meshgrid, this time spanned by x0- and x1-values. Lastly, we can give out the final parameter values and costs portrayed in the animations to ensure that we approximately visualized model convergence despite substantially restricting the number of epochs used to create the animations (see commented-out code!).
Additionally, we can also portray the path of gradient descent via a contour plot.
Finally, let me come back to why we need the cross-entropy (CE) loss function in logistic regression. First and foremost, it can be shown mathematically that the CE-cost function is convex with exactly one minimum, which is the global minimum. In contrast, applying the MSE-cost function on logistic regression results in a non-convex cost function.2 In the following, I will use a graphical approach to compare both methods on the example of our training data.
In Python, we introduce a new cost-function “MSE_cost( )” to quantify costs by the use of MSE. Like we did before with the cross entropy-cost function we can then create loss landscapes with respect to our training data (x_train2, y_train2) and our weights (w0,w1). I intentionally increased the range of possible values for our weights regarding the MSE-method to w0,w1 ∈ [-5,5] since this will help to illustrate the differences of both loss landscapes. Furthermore, numerical instability is not much of a concern with MSE as it is with CE.3
Obviously, the loss landscape on the right is looking “bumpier” now compared to the smooth and convex loss landscape of the cross-entropy cost function. For a more detailed view, we can try to visualize the MSE-loss landscape with a contour plot:
First, we can recognize that there is a minimum (“x”), located at roughly the same point where the CE-loss landscape had its global minimum (0.089,0.19). I will refer to this point, marked by “x” as the ‘global’ minimum of the MSE-loss landscape. I put ‘global’ in parenthesis because we lack the mathematical proof that this point actually is the global minimum of the MSE-loss landscape. Without further investigation, we could also assume that there is a local minimum in close proximity to the asterisk. With starting values for our weights within this area, gradient descent might get stuck and not converge to the ‘global’ minimum (“x”) in the middle of the contour plot. The following animation once more illustrates the full extent of non-convexity of the MSE-cost function above:
I will address gradient descent with the example of non-convex cost functions in more detail in my next article about neural networks.
I hope you found this article helpful. Should any questions arise or if you noticed any mistakes, feel free to leave a comment. The complete notebook can be found on my GitHub. Thank you for reading!