Data Preprocessing with scikit-learn — Missing Values | by Angelica Lo Duca | Towards Data Science

By popular demand from my previous article, in this tutorial I illustrate how to preprocess data using scikit-learn, a Python library for machine learning.
Data preprocessing transforms data into a format which is more suitable for estimators. Data preprocessing involves the following operations:
dealing with missing values
normalization
standardization
formatting
binning
In my previous articles I illustrated how to deal with missing values, normalization, standardization, formatting and binning with Python pandas. In this tutorial I show you how to deal with mising values with scikit-learn. For the other preprocessing techniques in scikit-learn, I will write other posts.
All the scikit-learn operations described in this tutorial follow the following steps:
select a preprocessing methodology
fit it through the fit() function
apply it to data through the transform() function.
The scikit-learn library works only with arrays, thus when performing every operation, a dataframe column must be converted to an array. This can be achieved through the numpy.array() function, which receives the dataframe column as input. In addition, the fit() function receives as input an array of arrays, each representing a sample of the dataset. Thus the reshape() function could be used to convert a standard array to an array of arrays.
You can download the full code from my Github repository.
In this tutorial we exploit the cupcake.csv dataset, which contains the trend search of the word cupcake on Google Trends. Data are extracted from this link. In the original dataset data in correspondence of 2004-02 and 2006-03 have been removed, in order to demonstrate how to deal with missing values. The original values were 5 and 10 respectively. We exploit the pandas library to import the dataset and we transform it into a dataframe through the read_csv() function.
import pandas as pddf = pd.read_csv('cupcake.csv')df.head(5)
Missing values are values not available in the original dataset. One solution to deal with missing values could be their removal from the dataset. However, this leads to data loss. The scikit-learn library provides two mechanisms to deal with missing values:
Univariate Feature Imputation
Multivariate Feature Imputation
Nearest neighbors imputation
In the Univariate Feature Imputation involves the replacement of missing values with a constant value or some provided statistics related to a feature. The SimpleImputer class can be used to perform univariate feature imputation. We specify which is the missing value through the missing_values parameter and the replacement strategy through the strategy parameter. For example, we can replace all the NaN values (identified by the numpy.nan variable) with the average value of the column.
import numpy as npfrom sklearn.impute import SimpleImputerpreprocessor = SimpleImputer(missing_values=np.nan, strategy='mean')
Now we can fit the preprocessor with the Cupcake column of the dataframe.
X = np.array(df['Cupcake']).reshape(-1,1)preprocessor.fit(X)
We apply the transform() function to our data.
X_prep = preprocessor.transform(X)
We convert it to the original shape by applying the inverse reshape() function and we store the result into a new column of the datafram df.
df['Cupcake_univariate'] = X_prep.reshape(1,-1)[0]df.head()
In the multivariate feature imputation each feature with missing values is calculated as a function of the other features. An iterative imputation is built thus the maximum number of iteration must be specified. We can use the IterativeImputer class. We consider two features: the column Cupcake and the index of the dataframe. Since the IterativeImputer is still at the experimental stage, we must enable it explicitly.
import numpy as npfrom sklearn.experimental import enable_iterative_imputerfrom sklearn.impute import IterativeImputerpreprocessor = IterativeImputer(max_iter=10, random_state=0)
We must convert the two features into arrays and transform them in the form [[f11,f21], [f12,f22] ...]. This can be done by applying the reshape() function to each feature and then the hstack() function as follows:
X1 = np.array(df['Cupcake']).reshape(-1,1)X2 = np.array(df.index).reshape(-1,1)X = np.hstack((X1,X2))
We fit the preprocessor with the obtained features.
preprocessor.fit(X)
And we apply the preprocessor to the same features X. In order to retrieve the result of the operation, we must apply the hsplit() function, which splits the array horizontally, then we apply the inverse reshape().
X_prep = preprocessor.transform(X)df['Cupcake_multivariate'] = np.hsplit(X_prep, 2)[0].reshape(1,-1)[0]
We can check results. Missing values are located at position 26 and 1. It is interesting to note that the two types of imputation produce different values.
df.iloc[26]
which gives the following output:
Mese                    2006-03Cupcake                     NaNCupcake_univariate      50.0792Cupcake_multivariate    28.6311Name: 26, dtype: object
and
df.iloc[1]
which gives the following output:
Mese                    2004-02Cupcake                     NaNCupcake_univariate      50.0792Cupcake_multivariate    21.6101Name: 1, dtype: object
This category of imputation fills missing values using the k-Nearest Neighbors approach. Each missing value is calculated using values from n_neighbors nearest neighbors that have a value. We can use the KNNImputer class of the scikit-learn library. In order to work properly, we must specify at least two features. Thus we exploit the X variable, previously defined.
from sklearn.impute import KNNImputerpreprocessor = KNNImputer(n_neighbors=5, weights="distance")preprocessor.fit(X)X_prep = preprocessor.transform(X)df['Cupcake_knn'] = np.hsplit(X_prep, 2)[0].reshape(1,-1)[0]
Now we can check results:
df.iloc[26]
which gives the following output:
Mese                    2006-03Cupcake                     NaNCupcake_univariate      50.0792Cupcake_multivariate    28.6311Cupcake_knn                10.7Name: 26, dtype: object
and:
df.iloc[1]
which gives the following output:
Mese                    2004-02Cupcake                     NaNCupcake_univariate      50.0792Cupcake_multivariate    21.6101Cupcake_knn             4.91892Name: 1, dtype: object
Definitely, the KNNImputer produces the nearest values to those in the original dataset (10 for position 26 and 5 for position 1).
In this tutorial we have learnt how to deal with missing values using the python scikit-learn library. Three basic classes exist to fill missing values: SimpleImputer, IterativeImputer, and KNNImputer. What is the best imputer? It depends on what you have to do.
If you wanted to learn how to perform the other aspects of data preprocessing using the scikit-learn library, stay tuned...
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and and Github.