Handle missing data with R: 10 daily used idioms | by Pavlo Horbonos | Towards Data Science

Data cleaning is one of the most time-consuming stages of the Data Analysis process. Many of its steps include acquainting the dataset, search of the missing values, their imputation or removing, and possibly a lot of repetitions of the same code lines but for different variables or their combinations. So, we have to search and accept any possibilities to speed the process up.
Some time ago I have presented the article about short Python idioms for missing values in datasets. Today I have prepared the compilation of similar scripts but in R language. Also, we will work again with the wines reviews dataset from Kaggle.
Yes, vectors in R start with 1, but before handling missing values we should have a dataset with missing values. Sometimes empty records are filled with substitutional strings like spaces (“ “), “empty”, “nan”, or some garbage. To start the work we should substitute them with “real” NA values. Lucky for us, we can do it at the stage of dataset opening:
wine_data <- read.csv(‘winemag-data-130k-v2.csv’,                       na.string = c(‘’, ‘i’, ‘P’))
na.string accepts a vector of substitutional values.
In the beginning, we need to check the presence of missing values. We have is.na() for these purposes. It receives data and returns a boolean vector.
Check if a single value missing:is.na(wine_data$region_1[2])
Get rows with missing elements in some columns:wine_data[is.na(wine_data$country),]
Check if there are NO missing values in a column:all(!is.na(wine_data$points))
Check if there is ANY missing value in a column:any(is.na(wine_data$country))
Previous scripts require some knowledge about the dataset and access to separate elements. But what if we want to see the whole picture? The answer is na.fail() function. It throws an exception if there is any missing value in whole data:
na.fail(wine_data)
There is a special function, which replaces a bunch of filters and indexes. It is mostly used with data frames and may be used to show all the rows with any missing value (in any column): complete.cases(). It returns a boolean vector and is very convenient for indexing.
Get the number of rows with missing datasum(!complete.cases(wine_data))
Get THE rows with missing datawine_data[!complete.cases(wine_data), ]
All previous scripts are mostly used for exploratory tasks: to show, to acquire, etc.. But what if we need to get the indexes of missing values? In this case, we should use which() function:
which(is.na(wine_data$country))#returns indexes of rows with missing data
It seems, that the only purpose of NA values existence is to be removed. Though, we may use them in some very interesting combinations. For example, we have a vector of mixed data:
test_vec <- c( 'ten', '5', 'n', '25', '10')
And we want to separate “convertible” strings (like ‘5’ or ‘25’) from the others. We can do it together with is.na() function:
test_vec[!is.na(as.numeric(test_vec))]# returns ( '5', '25', '10' )
NAs presence has some side effects — it turns the resulting value of almost any operation into NA. But there is a solution — a lot of R functions have built-in na.rm argument, which allows skipping missing values.
mean(wine_data$price)# returns NAmean(wine_data$price, na.rm = T)# returns the result
You can take a look at the previous scripts and think, that omitting of missing values is a very tedious job. Well, yes, it is. But in case if need to remove all NA at once and forget them as a nightmare — R has the next functions for us:
wine_data_totally_cleared_1 <- na.omit(wine_data)wine_data_totally_cleared_2 <- wine_data[complete.cases(wine_data),]
As you see — we can use as special function na.omit() as already known complete.cases(). The result is the same: there will be no more missing values in the dataset.
When we build some filters, we expect the result to have only matching values. It becomes especially valuable while working with data frames. But if you run, for example, this command:
wine_data_condition_NA <- wine_data[wine_data$price > 20,]
you will get not the correct subset only, but all the missing values too. It is the result of R data frame indexing, boolean arrays, and NA nature: missing value turns the condition result into NA which is treated as not FALSE. And we still have the pain of missing values search.
The solution is subset() function:
wine_data_condition <- subset(wine_data, wine_data$price > 20)
It returns the correct subset, but without NAs, which are filtered away.
Let’s return to the step when we looked at the whole picture of the dataset. We want to see the distribution of missing values, maybe some patterns, or dependency between variables. And we have another R package with the desired function — “mice” package. It contains functions, which walk through the dataset, count all the omissions, and build the picture of the dataset. For example, let’s take the sample: wine_data[1:5, 4:8]. It contains the next data:
As we see, there are some missing values. Let’s apply md.pattern() function from the mentioned package:
require(mice)md.pattern(wine_data[1:5, 4:8], plot = F)
And we get the table of patterns:
The leftmost index shows the number of rows. The rightmost index shows the number of missing features. The central matrix shows the patterns: which features combination is absent in the counted rows. So the first line tells us, that there are 2 rows with no absent data (0 missing features). And the third line shows, that there is 1 row in the data fragment with 1 missing feature — with price missing.
Now we are ready to visualize the distribution of missing values. We have another R package for this job — VIM library. Among many other functions it plots the distribution histogram of empty records and builds a visual representation of patterns table from the previous step:
require(VIM)aggr_plot <- aggr(wine_data, col=c(‘navyblue’,’red’), numbers=TRUE, labels=colnames(wine_data), ylab=c(“Histogram of missing data”,”Pattern”))
We have got a beautiful picture:
Now we can see, for example, that almost 60% of the rows have no region_2 value.
As for me, you should bring these code snippets to automatism. And you will gather significant analysis speeding. Also, as always, you may find the code with working examples on my GitHub:
github.com
Make sure, that you have seen my previous article about Python idioms: