Predict Missing Values in the Dataset | by Satyam Kumar | Towards Data Science

The real-world data often has a lot of missing values. The cause of missing values can be data corruption or failure to record data. The handling of missing data is very important during the preprocessing of the dataset as many machine learning algorithms do not support missing values.
There are various strategies to handle missing values in a dataset including the prediction of missing values. This article will cover understanding and implementation to handle missing values by predicting it using a Machine Learning algorithm.
Categories of columns having missing values:
Categorical (numerical, object)Continuous Variable (numerical)
Categorical (numerical, object)
Continuous Variable (numerical)
Categorical Variable:
Categorical Variable is a variable that can take on one of the limited, and usually fixed, number of possible values. A categorical variable can be a string or numeric type. Some of the possible examples of the categorical variable are:
Product Ratings: 1, 2, 3, 4, 5
Categorical rating: poor, satisfactory, good, excellent
Gender: Male, Female
Continuous Variable:
A continuous variable is one that can take on an uncountable set of values. For example, a variable over a non-empty range of the real numbers is continuous, if it can take on any value in that range.
To predict a categorical variable, a classification machine learning algorithm is required such as Logistic Regression, SVM, Naive Bayes, etc.
To predict a continuous variable, a regression machine learning algorithm is required such as Linear Regression, SVR, etc.
towardsdatascience.com
Prepare Dataset to Predict Missing Values:
Data used is Titanic Dataset from Kaggle
To predict the missing values in the dataset, create a new dataset such that:
new_data: columns except the column with missing valuestarget_class: column with missing value
In the above sample dataset image, Column Age, Cabin, Embarked contains null or missing values.
If we first want to impute the missing value of column Age, the new dataset to for training the model to predict missing values will be:
new_data: columns except the column "Age"target_class: column "Age"
Now we need to split the data into training data and data to predict, the rows which don’t have missing values in column Age will be declared as training data. The model trained from the above-formed dataset can be used to predict the missing values in the column Age.
training_data: rows from new_data where column "Age" does not have missing valuespredict_data: rows from new_data where column "Age" have missing valuestarget_class_traing: non-missing values in column "Age"
Correlation of Columns:
Do we need to train a model on the dataset with all the remaining columns? Sometimes training model with all remaining columns result to predict missing values better than some of the columns, but sometimes some of the columns which are highly correlated, a model trained of selective highly correlated columns tend to give a better result.
How to find a correlation between the columns?
Pandas library has a simple one-line implementation to find a correlation of all columns in the data frame.
data.corr()
The above image shows the Pearson Correlation coefficient values between each column. PCC measures linear correlation between two variables X and Y. It has a value between +1 and −1. A value of +1 is total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation.
The idea to compute the correlation between column Age and other columns to find the columns which have relation to the column Age. From the above correlation table, it is observed that 3 columns correlate with columns Age.
These columns are Pclass (PCC: -0.369226), SibSp (PCC: -0.308247), Parch (PCC: -0.189119).
Splitting of the dataset into Training and Prediction data:
Selecting 3 columns that are similar to column Age to form a dataset with 891 total rows.
Separating rows having null or missing values for prediction.
Preparing a Machine Learning model by training the X_train dataset and predicting the missing values in column Age.
Implementation:
Conclusion:
Every dataset has missing values that need to be handled intelligently to create a robust model. In this article, I have discussed the implementation to predict missing values. There is no thumb rule to handle missing values in a particular manner or using a particular model, the method which gets a robust model with the best performance. One can use the entire dataset for predicting missing values or some deleted columns. Having a domain knowledge about the dataset is important, which can give an insight into how to preprocess the data and handle missing values.
Thank You for Reading