Calculating and Setting Thresholds to Optimise Logistic Regression Performance | by Graham Harrison | Towards Data Science

I had learned about logistic regression, confusion matrices, ROC curves, thresholds etc. on the various data science courses I have undertaken but I never fully understood them and wanted to explore them in more detail.
Also, I had seen various examples online about how to recalculate the true and false classifications given a chosen threshold but these examples fell short of the detail I needed to put thresholds to work in the real world.
I suspected there might be a way to wrap threshold optimisation into a simple, object-oriented class so I could use them easily in future hence this article documents my learning journey to achieve these objectives.
Let’s start by importing the libraries we will need ...
We will also need some data to operate on. The data I selected was the bank credit default data from Kaggle — https://www.kaggle.com/c/credit-default-prediction-ai-big-data/overview.
I have performed some cleaning of that data including dealing with the null values, converting the categorical features and balancing the data to evenly represent the true and false classifications which is outside of the scope of this article. The data imported below includes all of those data cleaning steps ...
Let’s complete the preparation by splitting the data into testing and training so we can optimise against the training data and come back to the test data later on ...
((4754, 32), (1189, 32), (4754,), (1189,))
OK, let’s fit a basic LogisticRegression to the test data and then review the results ...
array([0, 0, 0, ..., 1, 0, 0], dtype=int64)
A visualisation of the confusion matrix will help us to evaluate the performance ...
The banking data is being used to predict customers who will go on to default against their credit agreements and those who will not default.
We can see straight away that a basic logistic regression has 79% accuracy having predicted 352 customers that will default and 3395 customers who will not.
That sounds pretty good right? Well if I was one of the managers taking delivery of this algorithm my immediate question would be — “what about the 934 false negatives?”. There are nearly a 1000 customers that the algorithm did not predict as defaulting who went on to default.
If the average cost of a default account to the bank is £10,000 my data science team has just cost the business £9.34m and I would definitely not be happy with that!
This simple analysis reveals three key points -
The accuracy measure may not always be the best way to evaluate the performance of a classification algorithm.
The way in which a classification algorithm is optimised is heavily dependent on what the business is trying to achieve.
The algorithm must be appropriately optimised to achieve the desired business outcomes.
A bit more digging will reveal all of the key metrics for the performance of our basic algorithm -
accuracy = 0.7881783761043332 precision = 0.8282352941176471
recall = 0.2737169517884914
f1 score = 0.4114552893045003
true positive rate (tpr) = 0.2737169517884914
false positive rate (fpr) = 0.02104959630911188
tpr-fpr = 0.2526673554793796
I would also usually take a look at the ROC (Receiver Operating Characteristic) curve -
The black dot represents one interpretation of the optimum point.
One measure that can be used is for calculating the optimum point on a ROC curve is TPR−FPR where TPR= True Positive Rate and FPR= False Positive Rate. The point at which the TPR−FPR is at its maximum value is the optimum point.
The graph is showing that if we set the threshold of the logistic regression to 0.31261 instead of the default which is 0.5 then we optimise the logistic regression algorithm for TPR−FPR.
If all this talk of thresholds is confusing there are many online articles that will explain the details, but the easiest way of envisaging it is this -
Each row of our data will be classified as Default=True or Default=False. The logistic regression assigns each row a probability of bring True and then makes a prediction for each row where that prbability is >= 0.5 i.e. 0.5 is the default threshold.
Once we understand a bit more about how this works we can play around with that 0.5 default to improve and optimise the outcome of our predictive algorithm.
From this quick analysis, the evidence suggests we can do better than accepting the default threshold of 0.5 that is implemented within the LogisticRegression algorithm that is part of the sklearn.linear_model library. For example, we already know that if we change that default to 0.31261 we will optimise for TPR−FPRTPR−FPR.
However, there is no easy way to change the threshold within the LogisticRegression class and typically data scientists will do this bit manually and duplicate effort across multiple projects.
What we need is a simple class that will do this for us every time we want to optimise the threshold.
Here is my first attempt at writing a class that extends the LogisticRegression class to optimise the threshold based on the TPR−FPRTPR−FPR calcualtion. It consists of a class definition, a method called threshold_from_optimal_tpr_minus_fpr and a predict method which over-rides the base class method enabling the threshold parameter to be passed in ...
Armed with this new utility class we can now easily fit the training data to it, ask the class to tell us the optimal threshold for TPR−FPRTPR−FPR and then use that threshold to return a new set of predictions ...
(0.3126109044627986, 0.3383474055618039)
Well, this optimisation has certainly changed the shape of the outcomes as shown by the confusion matrix. The false negatives and true negatives have reduced whilst the false positives and true positives have increased.
This finding led me to ask the question — “what other approaches are there to optimisation?”
Another way of evaluating the outcomes of a classification algorithm is to plot the precision and recall for the confusion matrix rather than the FPR and TPR.
As a quick reminder, precision and recall are calculated as follows -
I.e. Precision is the right hand column of the confusion matrix and precision is the bottom row.
Whereas TPR−FPR is commonly used to pick the optimum point on the ROC curve, the F1 score can be used to pick the optimum point on the precision-recall curve. The F1 Score is calculated as follows -
The F1 score is the harmonic mean of precision and recall. We use the harmonic mean instead of a simple average because it punishes extreme values.
The plot of precision vs. recall for our default algorithm looks like this -
Here we can see that the optimum (i.e. the maximum) F1 score is 0.5237148537347345 with the optimum precision and recall being 0.41 and 0.72 respectively. The threshold that would produce this outcome from the algorithm is 0.3126109044627986
This second attempt at extending the LogisticRegression class added a new method for optimising for the F1 score to produce the optimal precision and threshold values -
Now we can easily use the new version of the class to tell me what the threshold needs to be to optimise the f1 score and then to use that value to fine-tune the algorithm for the optimum precision and recall ...
(0.3126109044627986, 0.5237148537347345)
This version has significantly reduced the false negatives and represents the maximum possible optimisation of precision and recall.
Now imagine I took the new model back to the management team at the bank and they said “Well, that is an improvement but we absolutely have to have a recall of 90% for this model to be effective”.
Remember, recall is TP/(TP+FN) and at the moment this model has 922 / (364+922) = 71.7%.
We would need to explain to the management team that recall and precision are trade offs so if the model is tuned for 90% recall the precision must decrease but if that were acceptable, how could this be achieved?
This third attempt adds two new methods to calculate the required threshold for a given specific recall (and precision) which can now be used to achieve the performance requested by the management team -
(0.19142214362243234, 0.35382262996941893)
We can see that the recall requested by the management team has been achieved and that as a result the precision has been traded off from 41.3% to 35.4%.
At this point, I was getting rather carried away and I wanted a version of the optimisation that would implement a custom cost function.
Let’s suppose the management team now give us the following new information -
Every true negative means we can sell £10,000 of additional credit products to customers who will not default.
Every false positive costs us £1,000 because we could have engaged those customers but instead, we avoided them.
Every false negative costs us £1,500 because we failed to intervene and stop those customers defaulting.
Every true positive helps us to prevent credit default and each one generates £20,000 of new income.
A “cost function” to implement this can easily be constructed -
And with a bit more work we can modify our class to work out the threshold required to optimise this cost function and then apply it -
... which can be used as follows -
(0.4324142856743824, -40033500)
The new confusion matrix looks like this ..
... and here is the plot of our custom cost function values vs. the threshold values ...
Given that I had realised that the threshold for optimal accuracy could be calculated using a very similar approach to that for a custom cost function, I thought I would add that into the final version.
After all, many of the data science competitions do evaluate based on accuracy even though I had come to consider that it is not always, and in fact not often, the best measure of effectiveness of a classification algorithm.
That being said when I plugged the threshold for optimum accuracy calculated using the training data into the test data it lifted the accuracy measure of the predictions by a whole 1% and in a data science competition that might just make a difference!
The final version of the LogisticRegressionWithThreshold class adds a new method to calculate the threshold needed to optimize accuracy -
(0.5654501621491699, 0.7968026924694994)
Here is the confusion matrix for the optimal (highest) accuracy ...
... and here is the plot of accuracy vs. threshold ...
We have seen that there are many ways to optimise a logistic regression which incidentally can be applied to other classification algorithms. These optimisations include finding and setting thresholds for the optimisation of precision, recall, f1 score, accuracy, tpr — fpr or custom cost functions.
We have also seen that the chosen optimisation is heavily dependent on the desired business outcomes.
We have seen that the LogisticRegression class in sklearn.linear_model does not have a way to set the threshold so that the algorithm can be optimised but that we can use the LogististicRegressionWithThreshold class developed in this article to give us instant access to all the functionality necessary to tune the base algorithm according to the desired business outcomes.
The full code can be found on GitHub using this link -
github.com
If you enjoyed reading this article, why not check out my other articles at https://grahamharrison-86487.medium.com/?
Also, I would love to hear from you to get your thoughts on this piece, any of my other articles or anything else related to data science and data analytics.
If you would like to get in touch to discuss any of these topics please look me up on LinkedIn — https://www.linkedin.com/in/grahamharrison1 or feel free to e-mail me at GHarrison@lincolncollege.ac.uk.