Zero-Inflated Regression. Teach your regressor how to output... | by Dr. Robert Kübler | Towards Data Science

When working on regression problems, often you have target values that are continuously and evenly distributed in some range. Let me illustrate what I mean by this. Consider the following dataset 2-dimensional dataset:
In this case, the target values are all scattered in the range from about -300 to 400. Applying linear regression, we end up with the following model:
Nothing new, nothing fancy.
However, there are datasets that have an unusually high amount of zero targets in them. This is what we call a zero-inflated dataset.
Examples of such behavior are datasets with the target being the count of rare events. This includes the amount of
defects in manufacturing,
tornados or other natural disasters in a given date and place, or
crimes in some neighborhoods.
The common pattern is that usually, nothing bad happens, i.e. the target is zero. But occasionally, for example when a manufacturing machine malfunctions for some hours, the count skyrockets. In this case, the model actually has to predict something different from zero.
Many models can get distracted by such behavior, among them linear regression, support vector machines, and also neural networks. The chance that one of these models outputs a zero is quite small.
As an example, take a simple linear regression with one feature as depicted above. The output of the algorithm is a line that takes a value of zero exactly one time. The problem persists in higher dimensions, too. Or recall that neural networks can be expressed as a complex, nested formula — if you plug in random numbers, the chance is high that you won’t receive a zero as output.
But instead of talking about hypothetical scenarios, let us visualize the problem with a toy dataset.
This dataset consists of two columns:
the feature Age of a person and
the target Salary from working of this person.
In this made-up dataset, we can see that children and teens do not earn a salary. Starting at some age, money-making begins. People gain more experience and hence the salary increases with age. In the end, people retire and don’t earn a salary anymore. The story of life.
It is immediately clear that a simple linear regression will not work well here. Let’s do it anyway. 😎
Not a great fit. A line cannot capture all the variation in the data, but notice that if we would fit a line to the data between the ages of approximately 20 and 70, the fit in this interval would be great.
But this solves only part of the problem. We want a general-purpose model which is also able to make predictions for young and old people. Besides tree-based methods, let us take a look at a simple, yet effective approach to solve this problem.
We will now define an estimator that solves the problems related to zero-inflated datasets. Afterward, we use an already implemented version of it on our salary dataset.
The idea is quite simple. Consider the following two-step approach:
Train a classifier C that tells us whether the regression output is zero, or not.Train a regressor R on the part of the data with a non-zero target.
Train a classifier C that tells us whether the regression output is zero, or not.
Train a regressor R on the part of the data with a non-zero target.
If a data point goes into the model, the classifier first checks if the output should be zero. If yes, output zero. Otherwise, output the result of the regressor for this data point.
This is what we call a meta-model, i.e. a model that consists of other models. The great thing is that you can plug in any classifier and regressor you like.
Let’s get back to our salary dataset.
We see that the classifier has to capture the zero targets on both ends of the age axis. A simple logistic regression cannot do this (think about why!), but an SVM classifier with an RBF kernel might be sufficient.
The problem is that we have to implement this meta regressor now. While I believe that you could do this on your own, I want to point you to an awesome, already available implementation of this approach.
Note: I am the creator of this implementation and hence biased towards it.
It is part of the fabulous scikit-lego project that each of you should be aware of. This library contains a lot of useful scikit-learn compatible extensions, such as estimators, transformers, and more building blocks (hence the name). Let me quote from the project’s Github page:
“It was initiated by Matthijs Brouns and Vincent D. Warmerdam as a tool to teach people how to contribute to open source.”
Shout out to Vincent and Matthijs! So, take it to heart, check out this project and think about if you can contribute something, too. I already did.
Let’s finally code now.
import pandas as pdimport numpy as npfrom sklearn.svm import SVCfrom sklearn.linear_model import LinearRegressionfrom sklego.meta import ZeroInflatedRegressor# Create the salary dataset.np.random.seed(0)dataset = pd.DataFrame({'Age': np.random.randint(1, 100, size=200)})dataset['Salary'] = np.where((dataset.Age > 17) & (dataset.Age < 70), 1500*dataset.Age + 5000 + 10000*np.random.randn(200), 0).round()# Instantiate the zero-inflated regressor.zir = ZeroInflatedRegressor(    classifier=SVC(),    regressor=LinearRegression())# Easy fit!zir.fit(dataset[['Age']], dataset.Salary)
We can now plot the result:
This is exactly the behavior we want to see!
The only thing is that we have to deduct some points because the SVM misses a zero output at age 70. This in turn creates an outlier for the linear regression, which then tilts the line down a bit too much. Luckily, we can also use the ZeroInflatedReressor inside some hyperparameter optimization.
from sklearn.model_selection import GridSearchCVgrid = GridSearchCV(    estimator=zir,    param_grid={        'classifier__C': range(1, 10)    })grid.fit(dataset[['Age']], dataset.Salary)
You can get the best SVM parameter C via grid.best_params_ , which is 4 in our case. The result:
We have looked at datasets for regression that have a large number of zeroes as targets — zero-inflated data. This can unsettle many regressors such as support vector machines as well as neural networks.
As a way to circumvent this problem, we introduced the zero-inflated regressor, a meta-model that consists of a base classifier and a base regressor. The classifier checks whether the base regressor has to work, or if the prediction should be zero anyway. Otherwise, the base regressor takes over and outputs a number. Of course, this number can also be zero, but usually, it is not.
In some sense, the classifier makes the life of the regressor easier by keeping the regressor’s head free from clutter.
Finally, you might ask: “You mentioned tree-based models. What about them? They should be able to deal with zero-inflated datasets as well.” And yes, they can! Let us train a decision tree with a depth of 4 on the same dataset.
This also looks fine, but it has the usual step shape that we see always see with trees. This is not surprising though, as decision trees are basically only piecewise constant functions. So are random forests and gradient boosting algorithms because they are sums of trees.
The fit is good, but not as good as ours — a simple linear regression equipped with a support vector machine works better in our case. As usual, try out different models for your dataset at hand. Now you have another tool at your disposal that you can try out.
Have fun!
I hope that you learned something new, interesting, and useful today. Thanks for reading!
As the last point, if you
want to support me in writing more about machine learning andplan to get a Medium subscription anyway,
want to support me in writing more about machine learning and
plan to get a Medium subscription anyway,
why not do it via this link? This would help me a lot! 😊
To be transparent, the price for you does not change, but about half of the subscription fees go directly to me.
Thanks a lot, if you consider supporting me!
If you have any questions, write me on LinkedIn!