To Impute or not to Impute? A practical example when careless Imputation could lead to wrong conclusions | by Marie Stephen Leo | Towards Data Science

With great power to impute missing data, comes great responsibility for knowing what you’re doing.
In recent years, I’ve had the privilege of interacting with a lot of data professionals starting in the field and see many of them going through a similar epiphany:
What you learn in your coursework does not adequately prepare you for all the various scenarios you could encounter in the real world!
To be fair, there are just too many corner cases that no coursework can ever hope to cover comprehensively. Hence as a data professional, it’s up to every one of us to develop ourselves to the point where we can solve even the problems that we have never seen before. This post is about one such example that isn’t taught all that much in coursework but could mean the difference between an “average” data professional and an “exceeding-expectations” one.
Whenever missing data occurs in a modeling task, one of the commonly used techniques is to impute the missing data, which is necessary for our modeling code to work. However, in cases where there is a significant % of missing data such as high volume manufacturing measurements, the resultant model after imputation may seem to be weak on the surface. But just digging a little bit deeper could show that the model is rather strong, and the imputed values are messing with the metrics calculation. This post is about one such example to showcase this scenario using a typically heavily sparse dataset, which is Statistical Process Control measurement data. All the code for this post is available here: link
In a modern High Volume Manufacturing (HVM) facility, Statistical Process Control (SPC) systems are critical to ensure all the manufacturing equipment is delivering products that meet customer specifications. For example, in an automated car assembly facility, the customer specification could be that an average car engine should provide 22.5miles per gallon (mpg) of fuel and is the ‘response’ variable.
However, no process in nature will be on target 100% of the time. Instead, there would be normal variation causing some engines to deliver higher and some lower than 22.5mpg. Let’s say the customer understands this and is OK to tolerate a deviation of +/- 1.5mpg. Any engine beyond these tolerance limits cannot be sold to the customer and needs to be scrapped or sold for spare parts, causing a revenue loss to the company. Hence our response variable has a target of 22.5mpg, an upper control limit (UCL) of 24mpg, and a lower control limit (LCL) of 21mpg.
As expected, the senior management of the company wants 100% of the engines produced to meet these specifications (specs). However, again in nature, nothing but death and taxes are 100% certain. Instead, as Engineers and Statisticians, we settle for the next best thing: We can use SPC systems and 6-sigma processes to ensure 99.73% of engines will be within the specs.
Typical HVM product lines can take anywhere between a week to a few months to go from pure raw material to the complete finished product sold to the customer. Since the ‘response’ variables are customer specs, they can only be tested at the end of the manufacturing product line when the product is already built and ready to ship to the customer
The problem here is that if one of the equipment early on in the manufacturing line has gone rogue and produces defective parts, it can only be detected several weeks or months later when we test these products for the ‘response.’ Hence, a massive volume of products could be processed in the rogue equipment and cause a significant revenue impact to the company before it is detected. So obviously, this strategy is super risky.
To prevent substantial revenue losses from rogue equipment, HVM facilities continuously measure Critical Dimensions (CD) on the product at various steps in the manufacturing line. An SPC system then consumes these CDs to track whether the measurements are running within acceptable tolerance limits themselves following 6-sigma processes.
Now consider the following problem statement: All the CDs are performing within their designated 6-sigma tolerances, but the response variable has a large percentage falling outside the customer specs. We need to tighten the variation in some of the CDs to tighten the response. Hence there are two questions to be answered:
Considering all the CDs measured on the product, which has the most impact on the response’s variation and what are their contributions?How much should the variation of these relevant CDs be tightened to ensure the response meets customer specs?
Considering all the CDs measured on the product, which has the most impact on the response’s variation and what are their contributions?
How much should the variation of these relevant CDs be tightened to ensure the response meets customer specs?
Let us attempt to answer both these questions using data analytics. The first question shall be answered in this post, while the second question is a topic for my next post: keep a lookout here!
Disclaimer: All the data used in this study are entirely synthetically generated to demonstrate standard regression techniques.
Let’s consider the first question in more detail and have a look at the data available to us. As shown in the below snippet, our data consists of 200 feature columns, one response column, and 30K rows.
Dataset rows = 30000, columns = 201
Before we perform any analysis on this data, let’s first split the data frame into training and test sets so that there is a minimum possibility of information from the training set leaking to the test set. We split “train” as the first 80% (24K) rows and “test” as the last 20% (6K) rows to simulate using past data to train and future data to test. All subsequent study shall be performed on the “train” set and only use “test” set in the end to confirm our model’s performance on unseen data
X_train, y_train, X_test, y_test shapes = (24000, 200), (24000,), (6000, 200), (6000,)
Let’s first take a look at the distribution of our response variable to see if there is any problem worth solving in the first place. The response follows a nice normal looking distribution with ~10% of the response lying outside of the specs. So we cannot sell almost 10% of the products we are producing. What a waste! We can now appreciate why senior management is unhappy and internalize that we need to do something to improve this situation.
Next, let’s look at the features more closely. Our first observation is that there is a large number of “NaN” values indicating missing data in the dataset. We talk to our domain engineers to understand why is this so. The reason is pretty straightforward. In a manufacturing facility, any additional measurement takes up a finite amount of time without adding any value to the product. Hence, production engineers want to minimize the number of products measured to the least amount possible. For this product, the data tells us that, on average, ~80% of the products don’t get measured at each feature.
Mean missing = 81.39%; Min missing = 65.23% ; Max missing = 81.98%
But that’s ok, we know all about machine learning, and we can use imputation to make those missing values reappear magically, right? Maybe. Maybe not. Hold on to that thought until we discuss the results later on in this post. Let’s begin with imputing the “mean” of each column using the SimpleImputer class from sklearn.
# Impute missing valuesfrom sklearn.impute import SimpleImputerimputer = SimpleImputer(strategy='mean')
Our next observation is that each feature seems to be on a vastly different scale. To rank which features have the most impact on the response, we need to standardize them to the same scale. The standardization ensures that all of them are treated equally during our analysis. We shall achieve this using the StandardScaler class from sklearn.
# Scalingfrom sklearn.preprocessing import StandardScalerscaler = StandardScaler()
The problem statement posed to us is that we not only need to explain which features have an impact on the response, but we also need to provide an estimate of how much influence each has. Due to this interpretability requirement, we decide to ditch black-box models and stick with linear models that can easily be interpreted by engineering teams. We shall use LinearRegression, LassoCV, RidgeCV and ElasticNetCV regression estimators from sklearn to build and test the most popular linear models.
# Linear modelfrom sklearn.linear_model import LinearRegression, LassoCV, RidgeCV, ElasticNetCValphas = np.logspace(-6, -0.1, 100)lr_model = LinearRegression()lass_model = LassoCV(alphas=alphas, cv=3, max_iter=10000)rid_model = RidgeCV(alphas=alphas, cv=3)elas_model = ElasticNetCV(alphas=alphas, cv=3, max_iter=10000, l1_ratio=0.5)
Finally, to tie the Feature Engineering and the Modeling parts together, we use the Pipeline class from sklearn.
model_pipeline = Pipeline([        ('imputer', imputer),         ('scaling', scaler),        (model, estimator)        ])
For each estimator, we average the Root Mean Squared Error (RMSE) and R-squared (R2) over 10 Cross-Validation (CV) splits to evaluate their performance. Since Lasso, Ridge and ElasticNet have alpha Hyperparameter search embedded, we also print out the best alpha for each of these estimators. The results of all the models are bad. The RMSEs are only ~0.1 lower than the std dev of the response.
Now, it’s easy to stop at this point and tell senior management that the data is just too sparse (80% missing) and give up on the case altogether. But, a real data scientist doesn’t give up so quickly! We decide to dig into these results a little bit more to see if we can still extract something useful. After all, where there’s a will, there’s a way!
One of my favorite ways to visualize any regression model’s performance is to plot the Actual vs. Predicted Plot. In this case, we immediately see that there are two correlations. Some of the data points fall on a “Significant Correlation” line while many others fall on a “Weak Correlation” line. If we can eliminate those data points that fall on the Weak Correlation line, our performance metrics should show a nice improvement.
But what is causing this? Well, a big clue is in the predicted value of all these data points. It’s ~22.5, which is also the “mean” of our Actual Response data. If you recall, during the Feature Engineering section of this post, we imputed all missing data with the “mean” of each column. Since this data is highly sparse, those data points that have this “mean” data imputed across all the features will predict a response also close to the “mean”!
At this point, we can remove all the predictions that are equal to the “mean,” and that should improve our predictions significantly. However, to get the best predictive power possible, we first study what are the top features influencing the response. Since we Standardized all the features to the same scale, we can do this by simply ranking the absolute value of the standardized linear coefficients of all the features in descending order. Or in the case of Lasso regression, print all the non-zero standardized coefficients. Out of the 200 features in the dataset, five features are most correlated to the response variable from the Lasso model. With this information, we can now drop only the rows that contain imputed values in these five features used by the Lasso model to prevent noise from Imputation.
Finally, we test out the model on our test set by calculating the RMSE, R2, and plotting the Actual vs. Predicted plot. Recollect that we have dropped the rows having imputed data in any of the five features correlating to the response. As you can see, the model has very strong predictive power once the imputed values are removed during prediction! This suggests that while making predictions, we must limit ourselves to predicting only those rows that have data for all the 5 important features.
With this insight, we can influence the Engineering team who is in charge of choosing how to sample the measurements. For example, since each feature is measured on 20% of the rows, the Engineering team could implement a sampling strategy to ensure half of these measurements (10%) are measured on all 5 features and the other half (10%) remains random. Knowing your customers and using data to influence their decisions are important skills for any Data Professional!
In this post, we saw a practical example when Imputation may lead to a wrong conclusion by trying to answer the question: Considering all the CDs measured on the product, which has the most impact on the response’s variation and what are their contributions. However, digging a bit deeper could reveal a useful model and additional insights that could be used to influence customer behavior. In my next post, we shall see how to use this model to answer the second question: How much should the variation of the relevant CDs be tightened to ensure the response meets customer specs. All the code for this post is available here: link