Data Preprocessing with scikit-learn — Missing Values
By: Angelica Lo Duca
Tags: Python, Data Science, Machine Learning, Scikit Learn, Data Preprocessing

Data Preprocessing with scikit-learn — Missing Values

Image by Gerd Altmann from Pixabay

By popular demand from my previous article, in this tutorial I illustrate how to preprocess data using scikit-learn, a Python library for machine learning.

Data preprocessing transforms data into a format which is more suitable for estimators. Data preprocessing involves the following operations:

dealing with missing values

normalization

standardization

formatting

binning

In my previous articles I illustrated how to deal with missing values, normalization, standardization, formatting and binning with Python pandas . In this tutorial I show you how to deal with mising values with scikit-learn . For the other preprocessing techniques in scikit-learn, I will write other posts.

All the scikit-learn operations described in this tutorial follow the following steps:

select a preprocessing methodology

fit it through the fit() function

function apply it to data through the transform() function.

The scikit-learn library works only with arrays, thus when performing every operation, a dataframe column must be converted to an array. This can be achieved through the numpy.array() function, which receives the dataframe column as input. In addition, the fit() function receives as input an array of arrays, each representing a sample of the dataset. Thus the reshape() function could be used to convert a standard array to an array of arrays.

You can download the full code from my Github repository.

Data Import

In this tutorial we exploit the cupcake.csv dataset, which contains the trend search of the word cupcake on Google Trends. Data are extracted from this link. In the original dataset data in correspondence of 2004-02 and 2006-03 have been removed, in order to demonstrate how to deal with missing values. The original values were 5 and 10 respectively. We exploit the pandas library to import the dataset and we transform it into a dataframe through the read_csv() function.

import pandas as pd

df = pd.read_csv('cupcake.csv')

df.head(5)

Image by Author

Missing values are values not available in the original dataset. One solution to deal with missing values could be their removal from the dataset. However, this leads to data loss. The scikit-learn library provides two mechanisms to deal with missing values:

Univariate Feature Imputation

Multivariate Feature Imputation

Nearest neighbors imputation

Univariate Feature Imputation

In the Univariate Feature Imputation involves the replacement of missing values with a constant value or some provided statistics related to a feature. The SimpleImputer class can be used to perform univariate feature imputation. We specify which is the missing value through the missing_values parameter and the replacement strategy through the strategy parameter. For example, we can replace all the NaN values (identified by the numpy.nan variable) with the average value of the column.

import numpy as np

from sklearn.impute import SimpleImputer

preprocessor = SimpleImputer(missing_values=np.nan, strategy='mean')

Now we can fit the preprocessor with the Cupcake column of the dataframe.

X = np.array(df['Cupcake']).reshape(-1,1)

preprocessor.fit(X)

We apply the transform() function to our data.

X_prep = preprocessor.transform(X)

We convert it to the original shape by applying the inverse reshape() function and we store the result into a new column of the datafram df .

df['Cupcake_univariate'] = X_prep.reshape(1,-1)[0]

df.head()

Image by Author

Multivariate Feature Imputation

In the multivariate feature imputation each feature with missing values is calculated as a function of the other features. An iterative imputation is built thus the maximum number of iteration must be specified. We can use the IterativeImputer class. We consider two features: the column Cupcake and the index of the dataframe. Since the IterativeImputer is still at the experimental stage, we must enable it explicitly.

import numpy as np

from sklearn.experimental import enable_iterative_imputer

from sklearn.impute import IterativeImputer

preprocessor = IterativeImputer(max_iter=10, random_state=0)

We must convert the two features into arrays and transform them in the form [[f11,f21], [f12,f22] ...] . This can be done by applying the reshape() function to each feature and then the hstack() function as follows:

X1 = np.array(df['Cupcake']).reshape(-1,1)

X2 = np.array(df.index).reshape(-1,1)

X = np.hstack((X1,X2))

We fit the preprocessor with the obtained features.

preprocessor.fit(X)

And we apply the preprocessor to the same features X . In order to retrieve the result of the operation, we must apply the hsplit() function, which splits the array horizontally, then we apply the inverse reshape() .

X_prep = preprocessor.transform(X)

df['Cupcake_multivariate'] = np.hsplit(X_prep, 2)[0].reshape(1,-1)[0]

We can check results. Missing values are located at position 26 and 1. It is interesting to note that the two types of imputation produce different values.

df.iloc[26]

which gives the following output:

Mese 2006-03

Cupcake NaN

Cupcake_univariate 50.0792

Cupcake_multivariate 28.6311

Name: 26, dtype: object

and

df.iloc[1]

which gives the following output:

Mese 2004-02

Cupcake NaN

Cupcake_univariate 50.0792

Cupcake_multivariate 21.6101

Name: 1, dtype: object

Nearest neighbors imputation

This category of imputation fills missing values using the k-Nearest Neighbors approach. Each missing value is calculated using values from n_neighbors nearest neighbors that have a value. We can use the KNNImputer class of the scikit-learn library. In order to work properly, we must specify at least two features. Thus we exploit the X variable, previously defined.

from sklearn.impute import KNNImputer

preprocessor = KNNImputer(n_neighbors=5, weights="distance")

preprocessor.fit(X)

X_prep = preprocessor.transform(X)

df['Cupcake_knn'] = np.hsplit(X_prep, 2)[0].reshape(1,-1)[0]

Now we can check results:

df.iloc[26]

which gives the following output:

Mese 2006-03

Cupcake NaN

Cupcake_univariate 50.0792

Cupcake_multivariate 28.6311

Cupcake_knn 10.7

Name: 26, dtype: object

and:

df.iloc[1]

which gives the following output:

Mese 2004-02

Cupcake NaN

Cupcake_univariate 50.0792

Cupcake_multivariate 21.6101

Cupcake_knn 4.91892

Name: 1, dtype: object

Definitely, the KNNImputer produces the nearest values to those in the original dataset (10 for position 26 and 5 for position 1).

Summary

In this tutorial we have learnt how to deal with missing values using the python scikit-learn library. Three basic classes exist to fill missing values: SimpleImputer , IterativeImputer , and KNNImputer . What is the best imputer? It depends on what you have to do.

If you wanted to learn how to perform the other aspects of data preprocessing using the scikit-learn library, stay tuned…

If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and and Github.