Avoid Mistakes in Machine Learning Models with Skewed Count Data | by Mingjie Zhao | Towards Data Science

Count data is everywhere. Count data sounds so easy to deal with: they are just infinite integers, nothing special. If you think so, then you probably would handle them *wrong*. How so? This blog aims to provide you some tips for working with count data in Machine Learning (ML), to help you prevent some common mistakes that you may never have noticed before.
Let’s start with a simple question.
Suppose that we are developing ML models about movie watching, and there is a field called “count of cartoon movies that the user watched in the last 6 months”. Since everyone has a different taste of movies, we see values like 0, 1, 2, 3...101 (yes, the user who watched 101 movies must be a huge fan).
Now here’s the question for you:
What statistical distribution does this count data may follow?
If your answer is “normal distribution” or “I don’t know”, then congrats! I am sure this blog will help you.
If you don’t have much statistics knowledge, that’s fine. This blog aims to provide hands-on ML techniques, though I am also providing some details in statistics for readers who are curious to know.
First, let’s look at the plot and the data summary of this count data (a toy data created for demonstration). As we know, the fastest to dive into a brand new dataset is to make some plots. So here we go:
import numpy as npimport pandas as pdimport plotly.express as pxcount = np.concatenate((np.zeros(202),np.repeat(1, 50),np.repeat(2, 40)))np.random.seed(1)a=np.random.choice(range(3, 60),90)b=np.random.choice(range(61, 100),20)df = pd.DataFrame(np.concatenate((count, a, b, [101])),columns=['val'])fig = px.histogram(df, x='val',nbins=200)fig.show()
Now I am sure you would no longer assume it’s a normal distribution. And you probably notice that the data is...
...highly skewed with a long tail on the right, with a lot of 0s.
Actually, if we look at the median, half of them are 0s which indicates half of the users did not watch any cartoon movies in the last 6 months. You may start to agree with me (if you didn’t before) that this kind of data may need to be treated properly in your ML model. Yes, this type of data has a specific category in statistics: zero-inflated, indicating the distribution of the count data is sparse with many 0s (50% in this example).
Next, in section 2 we will discuss the common mistakes that newbies have when dealing with skewed count data as a modeling feature (i.e. independent variable). Then in section 3, we’ll see caveats when predicting the skewed count data is your goal (i.e. response variable). If you are ready, then let’s start!
We all know that the 1st thing to do after data exploration is data cleaning, where outlier removal is an important part. But please be careful when you apply some common outlier removal methods such as the Standard Deviation Method and Interquartile Range Method. As you see from the below, you probably would end up removing most of the values which are not outliers, since the median is 0 in this skewed data.
Mistake: use Standard Deviation Method -> the calculated percentage of outliers is: 96.5% (this method takes most of the data as outliers)
# Mistake 1: Standard Deviation Method# calculate summary statisticsdata_mean, data_std = np.mean(df.val), np.std(df.val)# identify outlierscut_off = data_std * 3lower, upper = data_mean - cut_off, data_mean + cut_off# remove outliersoutliers_removed = [x for x in df.val if x > lower and x < upper]print('the calculated percentage of outliers is:',100*len(outliers_removed)/df.shape[0])
Mistake: use Interquartile Range Method -> the calculated percentage of outliers is: 19.7% (better than Standard Deviation Method but still too many as outliers)
# Mistake 2: Interquartile Range Method# calculate interquartile rangeq25, q75 = np.percentile(df.val, 25), np.percentile(df.val, 75)iqr = q75 - q25# calculate the outlier cutoffcut_off = iqr * 1.5lower, upper = q25 - cut_off, q75 + cut_off# identify outliersoutliers = [x for x in df.val if x < lower or x > upper]print('the calculated percentage of outliers is:',100*len(outliers)/df.shape[0])
Solution for outliers removal: Be cautious with methods based on the data’s distribution statistics such as mean or median. We could consider using methods like percentile capping (remove points that are larger than a certain threshold such as 99%).
When performing linear regression models, people usually ignore the basic assumptions of using OLS models, like the error after modeling of all the independent variables should follow a normal distribution. Thus, the OLS model would not give you reliable results if you do nothing about the skewed count data.
Some statistics behind this if you are interested (or you could skip this):You don’t need to have independent variables follow normal distribution for linear regression, but why do you still care if your independent variables are skewed? The reasons are:
It’s highly possible that the skewed data would violate the assumption we mentioned above (normality of residual/homoscedasticity), and sacrifice the model accuracy, though it’s hard to tell before you actually plot the residuals.Heavy tails could increase the probability of having high leverage, which affects the performance of your regression model.It’s very common to calculate the t-statistics and p-value after running a linear regression, which aims to evaluate if the estimated coefficient is significantly different from 0. To make the inference valid, the distribution of estimated coefficients has to follow a normal distribution.
It’s highly possible that the skewed data would violate the assumption we mentioned above (normality of residual/homoscedasticity), and sacrifice the model accuracy, though it’s hard to tell before you actually plot the residuals.
Heavy tails could increase the probability of having high leverage, which affects the performance of your regression model.
It’s very common to calculate the t-statistics and p-value after running a linear regression, which aims to evaluate if the estimated coefficient is significantly different from 0. To make the inference valid, the distribution of estimated coefficients has to follow a normal distribution.
So what to do with it? Some people may think about using some data transformation methods. Well, this is the correct direction, but please be sure to avoid the next mistake:
Box-Cox is a great way to transform data but it should not be applied here also for the reason of assumption it holds: the data has to be all negative or all positive. With a large amount of 0 in our data, Box-Cox cannot be used anymore.
Solutions for data transformation:1) Add a small number (I would suggest 0.5) to all the values in this field and do log transformation. Why add 0.5? Because you cannot do log to 0, it’s undefined in mathematics.
2) Do a square root transformation by taking the square root of the values.
Both methods can help with the skewness, as shown in the output below.
print('skewness of raw data:', df['val'].skew())# log transformationdf['log_trans'] = np.log(df.val+0.5)print('skewness of log-tranformed data:', df['log_trans'].skew())# square root transformationdf['sqr_trans'] = np.sqrt(df.val)print('skewness of square root tranformed data:', df['sqr_trans'].skew())
Additional tips:1. If your data is really sparse such as >60% are 0s, I would suggest you change it as a binary variable:
df['binary'] = np.where(df.val>0, 1, 0)
2. If you are using non-linear ML methods such as tree models, you may not need to worry too much about having skewed data. However, linear regression models are widely used in industry for multiple reasons like interpretability, legislation requirements, etc.
When skewed count (zero-inflated) data is response variable, we may face a few similar issues like outlier removal, and you could refer to section 2 for the solutions. In terms of modeling the skewed count data, we would want to avoid the mistake:
Many people fit the data with linear regression (OLS) and find the model performance is not well. This is understandable since the data is NOT from the normal distribution (as discussed in section 1), so the OLS would not estimate it well (though it’s valid to do so).
To get better model performance, we need to understand that the data distribution under the hood. It’s back to the question we had at the beginning: “ What statistical distribution does this count data may follow?” The answer is:
Rather than describe it with a single distribution, it’s better to describe it as a binomial (watch or not) and a Poisson distribution (counts if watched any).
Theory suggests that the excess zeros are generated by a separate process from the count values, i.e. the positive values(ref. 1). Therefore, we could think the data has two parts: we could see the first part as a binary variable indicates if the value is 0 or not. And in the second part, we see the data come from a Poisson distribution for count data.
There are two ways you could try to fit the model:
If you have *some* 0s (< 30%):
You could try fit data with one model using the Zero-Inflated Regressor from Python library “scikit-lego”(ref 2)
If you have excess 0s:
Though you could still use the Python library “scikit-lego” as mentioned above, I personally would feel a bit concerned with so many zeros since it indicates an obvious segmentation/separation of the data. As we discussed the data comes from two processes, I would consider creating two ML models for this type of data:- The first model would predict if the value equals 0 or not. In the blog’s example, we use df[‘binary’] created in section 2 as the response variable for the first step. - Second, we filter out all the 0s and develop a model for the values larger than zero.
In our example of “watching cartoon movies”, we could consider this process like a “user segmentation” by predicting: 1) If users watched cartoon movies,
2) focus on users who watched cartoon movies and predict how many they’ve watched.
There are many studies that discuss zero-inflated data. Here are a few models you could try (Ref. 1):
Zero-inflated Poisson Regression
Zero-inflated Negative Binomial Regression — Negative binomial regression does better with overdispersed data, i.e. variance much larger than the mean.
Ordinary Count Models — Poisson or negative binomial models might be more appropriate if there are no excess zeros.
Count data is so common but they can be so difficult to handle with excess zeros. This blog provides you four common mistakes when cleaning, manipulating, and modeling the skewed count data. I hope you could learn from this blog so you feel more confident in your machine learning journey!
Note: the positive values I created in this example don’t necessarily follow Poisson distribution, since it’s just toy data for demonstration purposes. In reality, this kind of positive count data should follow the Poisson distribution.