Missing Values — Don’t Drop Them!
By: Brett Nebeker
Tags: Machine Learning, Data Science, Analytics, Data

As part of my hiring process, I have prospective data science candidates perform a data challenge. The data challenge asks the candidate to take the data provided, go through the model development process, and end with a classification model. Unless I’m hiring at a senior level, I’m not looking for the entire data science workflow to be perfect — at a high-level, I look for a thoughtful, logical approach to the problem at hand. This is usually exposed in the first 50 lines of code, where candidates are dropping features based on an arbitrary threshold of missing values. I’ve seen it at 40%, 80%, and a variety of points in between. I’ve also seen entire observations dropped because there was a single missing value (out of over 100 features). One candidate ended up dropping over 50% of his observations due to this…aggressive missing value treatment method.

Ultimately the threshold doesn’t matter, it’s the fact that we are dropping these features without even considering what the feature is, and what a missing value means. I actually ended up explicitly asking one candidate about this. I had around 10 straight data challenges returned, each challenge dropping every feature with 70%+ missing values. The rest of her challenge was pretty good, so I outright asked her, what in the world made you think this was a good idea without even exploring the feature? Her response was something along the lines of “I saw someone do it on the internet.” Again, these are junior level roles, I’m not looking for perfection. However, this is not the way to get a job with me — we may not agree on the right approach to the problem, but I need legitimate reasons why certain steps were performed.

Random vs Systematic Missing Values

The kind of missing values we are dealing with play a major role in how we treat them. It would be well worth your time to read about systematic and random missing values in the vtreat paper (section 2.1, but in reality, you should read the whole thing). Also, a fantastic side note, pyvtreat is now up on PyPi!

Anyway, what is the difference between systematic and random missing values, and more importantly, why does it matter?

Systematic: there is some logical reason why the value is missing. It could be that this feature is not applicable to the observation.

Random: the collection of the data failed at random, there is no specific business/process reason for the missing values. A couple of examples that could lead to random missing values are faulty sensors or broken processes.

These are two very different situations, and as such, should be treated differently. In cases where we have systematic missing values, that missing value means something to the overall process — it’s missing for a reason. This makes blanket methods an invalid approach — we should be thoughtful in how we approach missing values for each feature, and not just consider this proper treatment:

df.fillna(0, inplace=True)

Concrete Example

The example I always reference here is a feature in the Lending Club dataset, months since last delinquency. It’s somewhere around 65% missing. This feature is measuring how long it has been since a loan applicant had a delinquent account, which on the surface seems really important, but a lot of candidates end up dropping it! Thinking back to random vs systematic missing values, where would you guess this one falls? This is very likely a feature with systematic missing values — it is missing because it does not apply, or put more clearly, they haven’t had a delinquent account. Here are some treatments I’ve seen applied to this (and all other) features:

Fill with 0 — this tells our algorithm it has been 0 months since their last delinquency (meaning it happened basically yesterday). This gives our algorithm a ton of incorrect information and noise

Fill with the median — this is less terrible, but it’s really not accurate. Again, we are operating under the assumption that this is a systematically missing value. Filling with the median means that 50% of the population that has had a delinquency has a higher time since, and therefore might appear less risky. In reality, this will likely create a confusing nonlinear relationship for the algorithm to try to sort through, as the data on either side of the median will likely be riskier than the median.

Drop the observation/feature — read the title of the article!

A little information on the default rate and how it correlates with this feature. With low values, say less than a year, default rate is around 24%. As the time since delinquency increases, the value trends down, ending around 21%. The population without a delinquency hovers around 18% default rate. There is definitely some predictive power in this feature, and the above methods will not allow the feature to operate with the amount of predictive power it could. Instead, there are ways that we can fill this value to prevent the algorithm from getting lost in the noise.

Fill with a negative number — this would be helpful specifically for tree-based models, or other non-parametric approaches. The real value here should be ≥ 0, so a negative number splits this population into a separate category. But what if I’m dealing with linear models?

Fill with a large value — the longer ago the delinquency was, the less risky this customer is, so we can fill this with a large number that indicates to our model that this population is less risky than the rest of the population.

What if I Chose the Wrong Method?

All of this is somewhat of a guessing game. A lot of the time, it’s hard to truly know if these values are missing randomly or systematically, unless the data collection practices in your company are very robust and well-documented, and have a significant understanding of the context and how it relates to your analysis. So what happens if I chose the wrong approach? There’s one thing you should always do in an effort to give your algorithm complete information, in the event that you pick an incredibly terrible approach. That one thing (again, from the brilliant vtreat paper): create a new binary feature that is a flag for whether or not that value was originally missing. Your new feature will have a 1 if it was missing, and a 0 if there was some value there. For this specific feature, this is essentially a “have they ever had a delinquency” flag which can clearly be very helpful in your prediction.

Data science is a tough field, and requires candidates and practitioners alike to be logical and intelligent in the way they approach problems. One of the most challenging yet important parts of the data science workflow is handling messy data. The way you approach this piece shows your ability to conceptualize the problem at hand and come up with a robust solution, while providing huge lift in your model. The amount of lift from this step is almost as significant as feature engineering, but more on that later!

Questions? Feedback? Want to chat about data science and data strategy? Reach out! https://www.linkedin.com/in/brettnebeker/