Constrained Logistic Regression with Python | by Pararawendy Indarjo | Towards Data Science

Logistic regression is a basic yet popular classification model. Despite its simplicity, logistic regression is a powerful tool that is used in real-world contexts.
The method’s primary benefit is perhaps its explainability, thanks to the ease with which its parameters/coefficients may be interpreted. As a result, one can derive insights from the model by fully understanding the impact of each feature on the model.
In this blog, I’ll share how to apply specific restrictions to a logistic regression model. In other words, how to construct a logistic regression with given ranges on each of the feature’s coefficients.
The rest of the article will be arranged as follows.
The data used in this writingThe motivation: why we need to apply some constraints to the coefficients in the first place?The method: how to implement specific constraints to our logistic regressionThe consequences: acknowledge the effect of doing such constraining
The data used in this writing
The motivation: why we need to apply some constraints to the coefficients in the first place?
The method: how to implement specific constraints to our logistic regression
The consequences: acknowledge the effect of doing such constraining
Before we proceed further, note that all the codes involved in this article are available in my GitHub repo here.
We will use IBM telco customer churn data. This data is about a fictional telco company’s customer churn based on a variety of possible factors. The data is licenced under Apache Free Software License version 2 [1]. Therefore, we can use it for any purpose, modify, and distribute it [2].
The data can be downloaded from the IBM community website here.
We’ll perform several preprocessing steps as follows:
Convert column TotalCharges to float, by first removing rows whose values contain white space.For simplicity, drop categorical columns whose the number of distinct values > 2.Moreover, drop TotalCharges column to prevent multicollinearity issue with tenure (validate yourself 😉).
Convert column TotalCharges to float, by first removing rows whose values contain white space.
For simplicity, drop categorical columns whose the number of distinct values > 2.
Moreover, drop TotalCharges column to prevent multicollinearity issue with tenure (validate yourself 😉).
The above code should result in the following data snippet.
To make your life easier, I’ve put the clean/processed dataset on my GitHub here.
Suppose you’re tasked to build the classification model based on the dataset (re: predicting customer churn).
Because you’re an awesome data scientist, you’re aware that logistic regression is one of the methods available, so you create one with the wonderful sklearn library.
You report the following coefficients to your stakeholders, along with the relevant insights (re: interpretation of the coefficients).
The resulted coefficients:
In essence, your insights may go something like this:
The model suggests that gender male, SeniorCitizen, Partner, PaperlessBilling, and MonthlyCharges are associated with the increase of the likelihood the users to churn
While Dependents, tenure, and PhoneService are associated with the decrease of the likelihood of the users to churn (which is a good thing)
But, they (the stakeholders) challenge you with the following:
“The majority of your model’s output makes sense to me. I’m aligned that Dependents, tenure, and PhoneService should have negative coefficients (reducing the risk of churn).
However, based on my years of experience and knowledge, I firmly think that having a spouse (Partner) should also reduce the risk of churn, since the logic is similar to having dependents; you’ll require more calls to contact them, therefore you’ll be less likely to churn. As a result, the coefficient must NOT be positive (maximum zero).”
How would you accommodate this constraint?
clogistic is a Python implementation of the constrained logistic regression with a scikit-learn like API. Under the hood, the library uses convex optimizations to achieve this goal.
The first step is to materialize the constraints. That is, we specify the lower and upper bound of each feature’s coefficient. We will define the constraints as a pandas data frame for better interpretability.
From the aboveconstraint_df, we know that:
For feature gender, SeniorCitizen, PaperlessBilling, and MonthlyCharges, we do not impose any constraints, i.e. the coefficients of these features can be any real number (from super large negative to super large positive)
While for feature Partner, Dependents, tenure, and PhoneService, we require that the coefficients never be positive (maximum zero), as shown by upper bounds = 0.
Using this data frame, we can proceed with the model training using clogistic. The prerequisite step is, of course, to install the library (if you haven’t already). If you’re using Google Colab Notebook, simply run the following command.
!pip install clogistic
We’re now ready to train the model. The LogisticRegression API in clogistic is very similar to the one in sklearn, with the exception that we can specify a bounds argument while training the model, which accommodates the constraints we wish to apply.
The resulting coefficients are given below.
Look! The constraints are met nicely! Now feature Partner has a negative coefficient (virtually zero, though). Other three constrained features (Dependents, tenure, PhoneService) also remain negative. You may confidently return to your stakeholders with this result.
Next, for the sake of this tutorial, suppose that the constraints got updated as follows.
Partner, Dependents, tenure, PhoneService are still required to be non-positive
But now Dependents and PhoneService get lower bound requirement; Dependents > -0.2, PhoneService > -0.5
Don’t fret, all we need to do is to update the constraint data frame.
Afterwards, we do the same training step.
Retrieving the model coefficients comes next.
Notice that the updated constraints have been met by the model; We have Dependents coefficient = -0.2, and PhoneService coefficient = -0.5.
Horray! 🎉
One important point to note is that by imposing constraints, we are basically trading off some amount of optimality (model performance) in order to meet the constraints. As can be seen below, as we add additional constraints, our model’s performance in training data (F1 score) decreases.
F1 score on train set for sk_logreg model is 0.5593 F1 score on train set for cl_logreg model is 0.5575 F1 score on train set for cl_logreg_rev model is 0.5513
This occurs because the constraints would shrink the feasible region for the coefficient values eligible as the solution. As a result, the evaluation metric (here is F1 score) would be sub-optimal compared to the original (larger) feasible region without any constraints.
Congratulations for reading this far!
In this article, we learned how to build a logistic regression model with specified constraints on the coefficients. We accomplished this using clogistic Python library. Finally, we acknowledged the “side-effect” of imposing such constraints on our model’s performance.
As a friendly reminder, the whole notebook is available on my GitHub repo here for those who prefer the GitHub interface for these types of tutorials.
I hope this article helps when you encounter similar requirements! All in all, thanks for reading, and let’s connect with me on LinkedIn! 👋
[1] https://github.com/IBM/telco-customer-churn-on-icp4d#license. Remark: the dataset is used in this GitHub repository by IBM, which is the owner of the dataset. The whole repository is licensed under Apache Software License Version 2, hence also the dataset.