Logistic Regression: Bottoms-up Approach.
By: Hemanth Devarapati
Tags: Logistic Regression, Machine Learning, Metrics, Data Science, Gradient Descent

Logistic Regression: Bottoms-up Approach.

(Feature engineering ideology — a bonus)

Quick Question?

Can we use RMSE or MSE as an evaluation metric for classification? If yes, when can we use it? If not why can’t we use it?

I will disclose the answer in the end.

What is in it for you?

This article will walk you through the nuances of logistic regression and I will make you familiar with the feature engineering ideology. The first thing I want to clarify is logistic regression uses regression in its core but is a classification algorithm.

My confession:

This is my second favorite algorithm after decision trees merely for the simplicity and robustness it offers. Despite having a ton of classification algorithms out there this age out algorithm survived the test of time and you could see its seamless integration with the neural networks is what makes it very special.

Let’s talk data

To meet the objectives I have picked up a data set that will allow me to discuss feature engineering as well. It is a password strength classification dataset. It has 3 classes in it, 0- weak, 1- medium and 2- strong. It has only two columns which are password and it’s strength class. The dataset has around 700K records so be assured it is not a toy dataset.

Disclaimer!

This article expects a few probability concepts from your end. Please refresh probability (5 minutes) and odds (8 minutes). Read these excellent articles from BetterExplained: An Intuitive Guide To Exponential Functions & e and Demystifying the Natural Logarithm (ln). Then, review this brief summary of exponential functions and logarithms.

OMG! I don’t have the right data.

You can follow the code here.

Pitfall 1: In the password dataset it is possible to have commas(,) and you are reading CSV file which is a common separated value file. You might read more commas than you need and might land in an exception. So, you can sidewalk from this by setting error_bad_lines = False.

Distribution of passwords based on their strength

The data we have is a list of passwords, which pandas identify as ‘object type’. But, in order to work with an ML model, we need numerical features. Let’s generate new features out of the text we have. This is where feature engineering saves us. Feature engineering often works on intuition. There is no formulated right way of doing it. Usually, in industry people tend to rely on domain expertise in identifying new features. As per our data you are the domain expert. You know what are the key elements in an ideal password and we will start from there.

Created features:

No of characters in a password

No of numerals in a password

No of alphabets in a password

No of vowels in a password

No of consonants in a password (more of these means less meaning the password has)

New features. We will have to see if they actually contribute something.

Logistic Function

So we want to return a value between 0 and 1 to make sure we are actually representing a probability. To do this we will make use of the logistic function. The logistic function mathematically looks like this:

Let’s take a look at the plot

You can see why this is a great function for a probability measure. The y-value represents the probability and only ranges between 0 and 1. Also, for an x value of zero, you get a .5 probability and as you get more positive x values you get a higher probability and more negative x values a lower probability.

Make use of our data

Okay — so this is nice, but how the heck do we use it? Well we know we have five attributes — char_count, numerics, alpha, vowels, consonants— that we need to somehow use in our logistic function. One pretty obvious thing we could do is:

Where CC is our value for char_count, N is our value for numerics, A is our value for alpha, V is our value for vowels, CO is our value for consonants. For those of you familiar with Linear Regression this looks very familiar. Basically, we are assuming that x is a linear combination of our data plus an intercept. For example, say we have a password with a char_count of 8, numerics of 4, alpha of 4, vowels of 1 and consonants of 3. Some oracle tells us that 𝛽0=1, 𝛽1=2, 𝛽2=4, 𝛽3=6, 𝛽4=8 and 𝛽5=10. This would imply:

x= 1 + (2*8) + (4*4) + (6*4) + (8*1) + (10*3) = 95

Plugging this into our logistic function gives:

So we would give a 100% probability to a password with those features is medium (I took first row data)

Learning

Okay — makes sense. But who is this oracle giving us our 𝛽 values? Good question! This is where the learning in machine learning comes in :). We will learn about our 𝛽 values.

Step 1 — Define your cost function

For the sake of simplicity, we will remove one class and make it a binary classification problem. Later in the article, we will see how to do multi-classifications as well. For now, I’m mixing medium and weak as one class. If you have been around machine learning, you probably hear the phrase “cost function” thrown around. Before we get to that, though, let’s do some thinking. We are trying to choose 𝛽 values to maximize the probability of correctly classifying our passwords. That is just the definition of our problem. Let’s say someone did give us some 𝛽 values, how would we determine if they were good values or not? We saw above how to get the probability for one example. Now imagine we did this for all our password observations — all 700k. We would now have 700k probability scores. What we would hope is that for the strong passwords, the probability values are close to 1 and for the weak passwords the probability is close to 0.

But we don’t care about getting the correct probability for just one observation, we want to correctly classify all our observations. If we assume our data are independent and identically distributed, we can just take the product of all our individually calculated probabilities and that is the value we want to maximize. So in math, If we define the logistic function and x as:

This can be simplified to:

The ∏ symbol means to take the product for the observations classified as that password. Here we are making use of the fact that our data is labeled, so this is called supervised learning. Also, you will notice that for weak observations we are taking 1 minus the logistic function. That is because we are trying to find a value to maximize, and since weak observations should have a probability close to zero, 1 minus the probability should be close to 1. So we now have a value we are trying to maximize. Typically people switch this to minimization by making it negative:

Note: minimizing the negative is the same as maximizing the positive. The above formula would be called our cost function.

Step 2 — Gradients

So now we have a value to minimize, but how do we actually find the 𝛽β values that minimize our cost function? Do we just try a bunch? That doesn’t seem like a good idea…

This is where convex optimization comes into play. We know that the logistic cost function is convex — just trust me on this. And since it is convex, it has a single global minimum which we can converge to using gradient descent.

Here is an image of a convex function:

source: utexas.edu

Now you can imagine, that this curve is our cost function defined above and that if we just pick a point on the curve, and then follow it down to the minimum we would eventually reach the minimum, which is our goal. Here is an animation of that. That is the idea behind gradient descent.

So the way we follow the curve is by calculating the gradients or the first derivatives of the cost function with respect to each 𝛽. So let's do some math. First realize that we can also define the cost function as:

This is because when we take the log our product becomes a sum. See log rules. And if we define 𝑦𝑖 to be 1 when the observation is strong and 0 when weak, then we only do h(x) for strong and 1 — h(x) for weak. So let's take the derivative of this new version of our cost function with respect to 𝛽0. Remember that our 𝛽0 is in our x value. So remember that the derivative of log(x) is 1/𝑥, so we get (for each observation):

And using the quotient rule we see that the derivative of h(x) is:

And the derivative of x with respect to 𝛽0 is just 1. Putting it all together we get:

Simplify to:

Bring in the negative and sum and we get the partial derivative with respect to 𝛽0 to be:

Now the other partial derivatives are easy. The only change is now the derivative for 𝑥𝑖 is no longer 1. For 𝛽1 it is CC𝑖 and for 𝛽2 it is Ni. So the partial derivative for 𝛽1 is:

For 𝛽2:

Step 3 — Gradient Descent

So now that we have our gradients, we can use the gradient descent algorithm to find the values for our 𝛽s that minimize our cost function. The gradient descent algorithm is very simple:

Initially guess any values for your 𝛽 values

Repeat until convergence:

𝛽𝑖=𝛽𝑖−(𝛼∗ gradient with respect to 𝛽𝑖) for 𝑖=0,1,2,3,4,5 in our case

Here 𝛼 is our learning rate. Basically how large of steps to take on our cost curve. What we are doing is taking our current 𝛽 value and then subtracting some fraction of the gradient. We subtract because the gradient is the direction of greatest increase, but we want the direction of the greatest decrease, so we subtract. In other words, we pick a random point on our cost curve, check to see which direction we need to go to get closer to the minimum by using the negative of the gradient, and then update our 𝛽 values to move closer to the minimum. Repeat until converge means keep updating our 𝛽 values until our cost value converges — or stops decreasing — meaning we have reached the minimum. Also, it is important to update all the 𝛽 values at the same time. Meaning that you use the same previous 𝛽 values to update all the next 𝛽 values.

Gradient Descent Tricks

Normalize variables:

This means for each variable subtract the mean and divide by standard deviation.

Learning rate: If not converging, the learning rate needs to be smaller — but will take longer to converge. Good values to try …, .001, .003, .01, .03, .1, .3, 1, 3, …

Declare converges if cost decreases by less than 10^−3 (this is just a decent suggestion)

Plot convergence as a check

Follow the code for the implementation. After building the model from scratch we got an accuracy of 76% and I have used the sklearn package and got an accuracy of 99% this is pretty good.

Advanced Optimization

So gradient descent is one way to learn our 𝛽 values, but there are some other ways too. Basically these are more advanced algorithms that I won’t explain, but that can be easily run in Python once you have defined your cost function and your gradients. These algorithms are:

BFGS

L-BFGS: Like BFGS but uses limited memory

Conjugate Gradient

I will leave it here for you to explore. I would suggest not many know about these advanced optimization techniques rather they work well with gradient descent and its variants as they do a pretty decent job.

Evaluating Classification

Now that we have spent some time understanding the math behind logistic regression, let’s look more at how to evaluation classification problems. We can now fit a logistic regression model. Logistic regression can also leverage the regularization techniques we used in the linear regression post. They are used in exactly the same way — we penalize large coefficients by adding and L1 or L2 norm to our cost function.

Note we won't be demonstrating CV for hyper-parameter optimization here as we did with linear regression, but that is still applicable. In fact, almost all of the techniques we discussed in the previous post can also be used here. We will just be discussing new topics. We already know about the train and test data from the previous post. Let’s see how does the model do on test data.

Accuracy

One of the easiest metrics to understand when it comes to classification is accuracy: what fraction did you correctly predict.

Excellent!

One of the downfalls of accuracy is that it is a pretty terrible metric when you have imbalanced classes. Let’s look at our class balance:

Class Imbalance

So about 88% of our data are of the negative class — so if we just predicted everything to be negative, our accuracy would be about 88%! Not bad. Imagine other data where the negative class is only represented 1% of the time — you would get 99% accuracy by just predicting everything to be positive. That accuracy might feel good, but it is pretty worthless if that is your model.

Precision, Recall, and F1

To deal with some of these issues, often you will see precision, recall, and F1 used to evaluate classification tasks.

Precision is the number of true positives divided by all of the positive predictions (true positives plus false positives): Basically, how precise are our predictions when we predict something to be positive.

Precision

Recall is the number of true positives divided by all of the actual positives (true positives plus false negatives): Basically, what fraction of all the positives do we actually predict.

Recall

These two metrics trade-off between each other. For the same model, you can increase precision at the expense of recall and vice versa. Often, you have to determine which is more important to the problem at hand. Do we want our predictions to be precise or have a high recall?

What are true/false positives/negatives? Let’s take a look at a confusion matrix.

Confusion matrix

Here is our confusion matrix for our test predictions. The rows are the truth. So, row 0 are actual 0 labels and row 1 are actual 1 labels. The columns are the predictions. Thus, cell 0,0 counts the number of the 0 class that we got right. This is called true negatives (assuming we consider the label 0 the negative label). And cell 1,1 counts the number of the 1 class that we got right — true positives. Cell 0,1 are our false positives and cell 1,0 false negatives.

As you can probably tell, confusion matrices are extremely useful tools for error analysis. In this example, we have pretty symmetric errors — four missed within both classes. This is not always the case, and often there are more than two classes, so understanding which classes get confused for others can be very useful in making the model better.

Sometimes you just want a balance between precision and recall. If that is your goal, F1 is a very common metric. It is the harmonic mean of precision and recall:

Harmonic mean of precision and recall

The harmonic mean is used because it tends strongly towards the smallest element being averaged. If either precision or recall is 0 then your F1 is zero. The best F1 score is one.

Precision / Recall Curves

With all of the previous metrics, we have been using binary predictions. But really logistic regression returns probabilities, which are extremely useful.

Now instead of a single 1/0 number for each prediction, we have two numbers: the probability of being class 0 and the probability of being class 1. These two numbers must sum to one so 1-probability of 1 = probability of 0. A useful thing to do is to make sure these probabilities actually seem to correlate with the classes:

What this plot is showing us is that for our training data that indeed our positive class tends to have a high probability of being positive and our negative class has a high probability of being negative. This is basically what we designed our algorithm to do, so this is somewhat to be expected. It would also be good to check that this is true for validation sets.

Sklearn’s predict function just takes the class with the highest probability as the predicted class, which isn’t a bad method. One might want, though, a very precise model for positive predictions. One way we could do this is only call something positive if the probability is over 90%. To understand the trade-offs between precision and recall with different cut-off values, we can plot them:

Nice! We can clearly see the trade-offs between P and R as we adjust our thresholds. And in fact, we can find the cut-off that maximizes F1 (NOTE: we are using testing data here, which would not be good in practice. We would want to use a validation set to choose a cut-off value)

ROC Curve

ROC curves are another popular method for evaluating classification tasks. It is a plot with the recall value (or true positive rate) on the y-axis and the false positive rate (false positives divided by all the actual negatives) on the x-axis. Thus, it shows how your model trades off between these two values. The closer your value is to the top-left corner of the graph, the better. Let’s take a look for our test data:

ROC curve USP: It would work even with highly imbalanced data

Clearly, our model has done very well. The AUC (area under the curve) score you see in the bottom right is the area under the ROC curve, a value of 1 being the best one can do. We got to one. This curve can also be used to pick a threshold value if you know the trade-off you want to make between TPR and FPR.

Multi-class

Logistic regression can also handle more than 2 classes. There are two ways we can do this:

One-vs-rest method: With this method, we could train a classifier per class where the positive values are when that class is active and the negative values for all other classes.

Cross-entropy loss: We can also change our loss function to incorporate multiple classes. The most common way to do this is by using the cross-entropy loss:

Cross entropy loss

This loss takes the negative average of the predicted probabilities for the correct class. Thus, optimizing it attempts to get the predicted probabilities for the correct class as high as possible.

Interpreting logistic regression

Interpreting the coefficients of logistic regression is a little trickier than that of linear regression. Let’s take a look at our coefficients:

Our most positive coefficient is on char_counts with a value of 5 (roughly). Since we used a log-loss, we need to convert this value by doing e^5 which is 147.52. That is for every 1 unit increase in char_counts the odds of the password being a strong one increases by 14752% because we now have converted to an odds ratio. For more details on this, check out this post.

Answer to the question

I would like to show it using an example. Assume a 2 class classification problem and 6 instances.

Assume, True probabilities = [1, 0, 0, 0, 0, 0]

Case 1: Predicted probabilities = [0.2, 0.16, 0.16, 0.16, 0.16, 0.16]

Case 2: Predicted probabilities = [0.4, 0.5, 0.1, 0, 0, 0]

The MSE in the Case 1 and Case 2 is 0.128 and 0.1033 respectively.

Although Case 1 is more correct in predicting the class for the instance if 0.5 is considered the threshold, the loss in Case 1 is higher than the loss in Case 2. Why is this? MSE is a metric that measures the standard deviation of the predictions from the ground truth. And also it’s application might be limited only to binary classification. It still might be used in specific instances. The RMSE is one way to measure the performance of a classifier. Error-rate (or No. of misclassification) is another one. If our goal is a classifier with low error-rate, RMSE is inappropriate and vice versa. This explains why we don’t use MSE/RMSE as an evaluation metric for classification problems. Other metrics provide more flexibility and understanding of the classifier performance than an MSE.

Code: Github

My article about linear regression

If you need any help you can get in touch: LinkedIn

References:

Hands-on machine learning by Geron Aurelien