The handling missing values exclusive pythonic guide | by Philippe Bouaziz,Data Scientist at Africa4Data,PhD | Towards Data Science

Handling missing values is a key step in exploratory data analysis (EDA) for most data science projects whether your developing machine learning models or business analytics. Most libraries including scikit-learn don’t build a model using data with missing values. Due to the high quantity of data, finding tricks for getting the best imputing values results is a massive advantage for becoming a unicorn data scientist. In this article, we will review the 3 most successful open source short python code lines which can be combined for handling missing values.
For this article, we will be analyzing the samples flowers, titanic, and house prices Kaggle datasets you can find here.
There are many scenarios dealing with missing values, missing numbers are commonly represented in python as nan which is short for “not a number”. The classical method consists in detecting cells with missing values, and count their numbers in each column with this command:
missing_val_count_by_column = (data.isnull().sum())print(missing_val_count_by_column[missing_val_count_by_column > 0
3 mains methods can be found in most Kaggle competitions, let us review them:
One way to drop columns with missing values is to drop the same columns in both train/test dataframe , as show below :
From original data-frame:
data_without_missing_values = original_data.dropna(axis=1)
With train, test data-frame:
cols_with_missing = [col for col in original_data.columnsif original_data[col].isnull().any()]reduced_original_data = original_data.drop(cols_with_missing, axis=1)reduced_test_data = test_data.drop(cols_with_missing, axis=1)
After deleting your missing values your model loses access to this column which results in losing essential data. However, this strategy can be appropriate when most values in a column are missing.
A better option than dropping your datasets columns is filling the missing values with the data distribution mean, as show below:
from sklearn.impute import SimpleImputermy_imputer = SimpleImputer()data_with_imputed_values = my_imputer.fit_transform(original_data)
This option is integrated commonly in the scikit-learn pipelines using more complex statistical metrics than the mean. A pipelines is a key strategy to simplify model validation and deployment.
The last strategy systematically filling missing values randomly. If you need to get the best results, you need to consider your column statistical metrics. Here’s how it might look:
# make a copy to avoid changing original data (when Imputing)
new_data = original_data.copy()
# make new columns indicating what will be imputed
cols_with_missing = (col for col in new_data.columnsif new_data[col].isnull().any())for col in cols_with_missing:new_data[col + ‘_was_missing’] = new_data[col].isnull()
# Imputation
my_imputer = SimpleImputer()new_data = pd.DataFrame(my_imputer.fit_transform(new_data))new_data.columns = original_data.columns
In most cases this approach will improve your results significantly.
Now let us practice with the most known Kaggle datasets. We will review examples from titanic, flower, price housing Kaggle competitions datasets.
To master missing value handling, I advise you to follow each step using this notebook and repeat the same steps with your datasets.
data = pd.read_csv(‘../input/titanicdataset-traincsv/train.csv’)
from sklearn.ensemble import RandomForestRegressorfrom sklearn.metrics import mean_absolute_errorfrom sklearn.model_selection import train_test_splittitanic_target = data.Survivedtitanic_predictors = data.drop(‘Survived’, axis=1)
# For the sake of keeping the example simple, we’ll use only numeric predictors.
titanic_numeric_predictors = titanic_predictors.select_dtypes(exclude=[‘object’])X = titanic_numeric_predictors.select_dtypes(exclude=[‘object’])y = titanic_target
# Divide the data in train and test datasets.
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)
Method 1
cols_with_missing = [col for col in X_train.columnsif X_train[col].isnull().any()]reduced_X_train = X_train.drop(cols_with_missing, axis=1)reduced_X_test = X_test.drop(cols_with_missing, axis=1)print(“Mean Absolute Error from dropping columns with Missing Values:”)print(score_dataset(reduced_X_train, reduced_X_test, y_train, y_test))
Mean Absolute Error from dropping columns with Missing Values: 0.391340782122905
Method 2
my_imputer = SimpleImputer()imputed_X_train = my_imputer.fit_transform(X_train)imputed_X_test = my_imputer.transform(X_test)print(“Mean Absolute Error from Imputation:”)print(score_dataset(imputed_X_train, imputed_X_test, y_train, y_test))
Mean Absolute Error from Imputation : 0.3914525139664804
Method 3
imputed_X_train_plus = X_train.copy()imputed_X_test_plus = X_test.copy()cols_with_missing = (col for col in X_train.columnsif X_train[col].isnull().any())for col in cols_with_missing:imputed_X_train_plus[col + ‘_was_missing’] = imputed_X_train_plus[col].isnull()imputed_X_test_plus[col + ‘_was_missing’] = imputed_X_test_plus[col].isnull()
# Imputation
my_imputer = SimpleImputer()imputed_X_train_plus = my_imputer.fit_transform(imputed_X_train_plus)imputed_X_test_plus = my_imputer.transform(imputed_X_test_plus)print(“Mean Absolute Error from Imputation while Track What Was Imputed:”)print(score_dataset(imputed_X_train_plus, imputed_X_test_plus, y_train, y_test))
Mean Absolute Error from Imputation while Track What Was Imputed: 0.38586592178770945
=> The best method is: Method 1
data = pd.read_csv(‘../input/flower-type-prediction-machine-hack/Train.csv’)
from sklearn.ensemble import RandomForestRegressorfrom sklearn.metrics import mean_absolute_errorfrom sklearn.model_selection import train_test_splitflower_target = data.Classflower_predictors = data.drop(‘Class’, axis=1)
# For the sake of keeping the example simple, we’ll use only numeric predictors.
flower_numeric_predictors = flower_predictors.select_dtypes(exclude=[‘object’])X = flower_numeric_predictors.select_dtypes(exclude=[‘object’])y = flower_target
# Divide the data in train and test datasets.
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)
Method 1
cols_with_missing = [col for col in X_train.columnsif X_train[col].isnull().any()]reduced_X_train = X_train.drop(cols_with_missing, axis=1)reduced_X_test = X_test.drop(cols_with_missing, axis=1)print(“Mean Absolute Error from dropping columns with Missing Values:”)print(score_dataset(reduced_X_train, reduced_X_test, y_train, y_test))
Mean Absolute Error from dropping columns with Missing Values: 1.316111627552926
Method 2
my_imputer = SimpleImputer()imputed_X_train = my_imputer.fit_transform(X_train)imputed_X_test = my_imputer.transform(X_test)print(“Mean Absolute Error from Imputation:”)print(score_dataset(imputed_X_train, imputed_X_test, y_train, y_test))
Mean Absolute Error from Imputation: 1.3137274481525667
Method 3
imputed_X_train_plus = X_train.copy()imputed_X_test_plus = X_test.copy()cols_with_missing = (col for col in X_train.columnsif X_train[col].isnull().any())for col in cols_with_missing:imputed_X_train_plus[col + ‘_was_missing’] = imputed_X_train_plus[col].isnull()imputed_X_test_plus[col + ‘_was_missing’] = imputed_X_test_plus[col].isnull()
# Imputation
my_imputer = SimpleImputer()imputed_X_train_plus = my_imputer.fit_transform(imputed_X_train_plus)imputed_X_test_plus = my_imputer.transform(imputed_X_test_plus)print(“Mean Absolute Error from Imputation while Track What Was Imputed:”)print(score_dataset(imputed_X_train_plus, imputed_X_test_plus, y_train, y_test))
Mean Absolute Error from Imputation while Track What Was Imputed: 1.3199202727466344
=> The best method is: Method 1
data = pd.read_csv(‘../input/house-prices-advanced-regression-techniques/train.csv’)
from sklearn.ensemble import RandomForestRegressorfrom sklearn.metrics import mean_absolute_errorfrom sklearn.model_selection import train_test_splithouse_target = data.SalePricehouse_predictors = data.drop(‘SalePrice’, axis=1)
# For the sake of keeping the example simple, we’ll use only numeric predictors.
house_numeric_predictors = house_predictors.select_dtypes(exclude=[‘object’])X = house_numeric_predictors.select_dtypes(exclude=[‘object’])y = house_target
# Divide the data in train and test datasets.
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)
Method 1
cols_with_missing = [col for col in X_train.columnsif X_train[col].isnull().any()]reduced_X_train = X_train.drop(cols_with_missing, axis=1)reduced_X_test = X_test.drop(cols_with_missing, axis=1)print(“Mean Absolute Error from dropping columns with Missing Values:”)print(score_dataset(reduced_X_train, reduced_X_test, y_train, y_test))
Mean Absolute Error from dropping columns with Missing Values: 16907.894178082195
Method 2
my_imputer = SimpleImputer()imputed_X_train = my_imputer.fit_transform(X_train)imputed_X_test = my_imputer.transform(X_test)print(“Mean Absolute Error from Imputation:”)print(score_dataset(imputed_X_train, imputed_X_test, y_train, y_test))
Mean Absolute Error from Imputation: 17433.810719178084
Method 3
imputed_X_train_plus = X_train.copy()imputed_X_test_plus = X_test.copy()cols_with_missing = (col for col in X_train.columnsif X_train[col].isnull().any())for col in cols_with_missing:imputed_X_train_plus[col + ‘_was_missing’] = imputed_X_train_plus[col].isnull()imputed_X_test_plus[col + ‘_was_missing’] = imputed_X_test_plus[col].isnull()
# Imputation
my_imputer = SimpleImputer()imputed_X_train_plus = my_imputer.fit_transform(imputed_X_train_plus)imputed_X_test_plus = my_imputer.transform(imputed_X_test_plus)print(“Mean Absolute Error from Imputation while Track What Was Imputed:”)print(score_dataset(imputed_X_train_plus, imputed_X_test_plus, y_train, y_test))
Mean Absolute Error from Imputation while Track What Was Imputed: 17404.84109589041
=> The best method is: Method 1
If you have some spare time I’d recommend, you’ll read this article :
medium.com
Refer to this link for the complete missing data analysis of these datasets using these methods.
This brief overview is a reminder of the importance of using more than one method for handling missing values using python in data science. This post has for the scope to covered 3 essential Python handling missing values for making a complete exploration workflow, as well as useful documentation.