Statistical test for MCAR in python…
By: Sindhuja Hariharan
Tags: Data Science, Python, Handling Missing Values, Data Analysis, Statistical Analysis

Have you wondered if missing “age” is related to the “Salary” of the respondant in a survey? Have you ever thought of analysing associations between various missing values in a dataset? How can you be sure that the absent data are because of no definite pattern? — Answer to these questions is fairly straight forward — This article shows you how to statistically prove the relationships between variables where data is missing — in python.

Picture Source: unsplash

Ever since I came across LittleMCAR’s test in R for finding out the significance of the association between the missingness of variables, I have been searching for a similar test in python. This article will shed some light on performing a similar test in python. In this article, I will walk you through a set of codes, which analyses on missing values in categorical data.

Before deep-diving into testing missing data mechanisms, let’s just understand them clearly.

Let’s say we have a dataset, and we need to analyze those columns for missing values as well as for the relationship between variables. By analyzing the mechanisms, we can better decide how we want to handle them.

There are three different mechanisms by which data goes missing in any dataset.

1. Missing completely at random: (MCAR)

2. Missing at Random: (MAR)

3. Missing not at Random: (MNAR)

MCAR:

What does it mean? When missing data points do not follow any particular reasoning or a pattern. For example, you have the demographic data of residents in your community. But there are 50% missing values under the variable “Middle Name”. This 50% of data is a perfect example of data MCAR. There is no pattern or a specific reason behind the middle name being blank in most of the entries.

How to deal with it? The following methods can be adopted in case of data MCAR.

a. List-wise deletion: Deleting the record if the dataset has missing data in any of its variables/columns. This works best only when the missing data is small in number, like when in a dataset only 2% of the data is missing completely at random.

b. Pair-wise deletion: Pairwise deletion only deletes cases when one of the variables being used in the statistical method that is considered, is missing.

Internally, it works along the same lines of a correlation matrix. In case of missing values between two variables, (in pairwise-sense) finding a correlation matrix takes all the complete cases for those two variables into account. Let’s say the number of cases in this scenario is N.

When another set of variables is taken and the correlation matrix is calculated, the number of complete cases would differ from N.

This serves as the major difference between Listwise and Pairwise deletion.

Pairwise deletion has the advantage of minimal data loss. When there is a dataset that has widespread missing values in almost all variables, pairwise would be a wiser choice of missing value treatment.

c. Mean, median & mode imputation: Missing values can also be replaced with the mean, median, and mode values of the respective variables.

MAR:

What does it mean? When missing data points follow a pattern. Let’s take the same example of the demographic data of residents in your community. But this time the salary of few men who have ages greater than 45 are missing.

Here the missingness in data is attributed to another variable’s data. Hence, it follows a “missing at Random’ mechanism. MAR is probably most tricky to understand because of its name.

How to deal with it? Since there is a relationship involved in this mechanism, the best option here would be to go for an imputation technique — mean, median, mode, or multiple imputations — (This will be the scope of this article)

MNAR:

What does it mean? When missing data points follow a pattern with its data, it is meant to follow an MNAR mechanism. For example, in the same demographic data of residents in your community, let’s say the salary of a few men is missing when salary is beyond a certain amount. (say a million)

Here, it follows a “Missing Not at Random’ mechanism. Typically, when data missing is not MCAR or MAR, it tends to follow MNAR.

How to deal with it? Since there is a self-induced relationship in MNAR’s case, the best way to avoid them is to do some data collection or model the missing data.

The reason why we need to analyze the missing value mechanisms is that we need to understand the best way to impute the missing value so that the accuracy of our ML models stays uninfluenced.

Let’s get coding now…

I have taken this Kaggle dataset for the analysis. Let’s first pull this data into a data frame and check the total number of missing values within the dataset. I have dropped the timestamp variable from the dataset for now, since my goal here is to work with missing values and their relationships. I have also replaced the gender column values with minimal categories — “Male” and “Female”.

As we can see in the below screenshot, there are missing values in a few of the columns — State, Self_employed, work_interfere, and comments have a lot of missing data.

Now, there is just one numerical variable in the chosen dataset, but there are no missing values in it. So, let’s just separate the categorical variables in a separate data frame.

Now that we have separated the categorical variables with complete and incomplete cases, we need to analyze the association between each variables’ complete and incomplete cases, using traditional chi-sq. test for independence and the Cramer’s coefficient.

The Chi-Square test of independence is used to determine if there is an association between two categorical variables. A contingency table or a crosstab is used by the chi-sq. test to check for the frequency in those categories.

The Cramer’s coefficient is a measure of association between two nominal variables, giving a value between 0 and +1.

Source: Wikipedia

Source: Wikipedia

The following function calculates the Cramer's coefficient value for all the variable combinations in the dataset.

Source: Stack Overflow

After applying both chi-squared and Cramer’s logic into our data frame, we get the following as an output.

As we can see above, for the complete cases, the Cramer coefficient is the highest when a column is compared with itself. The columns — “Independent” show the result of the chi-sq test significance. If the p-value for chi-sq was <0.05, the association is going to be dependent.

We perform the same analysis for incomplete cases too.

Here, Cramer’s Coefficient is Nan for few rows because the number of incomplete cases for a few of the variables could have been 1.

Generally, the accepted category for any Cramer’s coefficient association is taken from a standard set of intervals.

Depending on the above criteria, let’s segregate our output into four different data frames –

1. cramer_high

2. cramer_moderate

3. cramer_low

4. cramer_little_if_any

The following are the variables which have high association among themselves.

1. Self-employed & No-Employees

2. Treatment & work_interfere

3. Seek_help & wellness_program

Among these, “Self-employed & work_interfere” has missing data in our dataset.

The cramer’s V for incomplete cases, tells us that among the missing data, there is a strong relationship between “Self_employed” & “no_of_employees” variables ( >0.5).

Same is the case with our other variable which has missing data — “work_interfere”. Whether a person is under treatment for a mental illness or not, greatly influences the way the illness interferes in his work during the day.

However, when you look at the chisq test’s results for these two variables, there is no difference in conclusion for the complete and incomplete cases. This means that the association can’t be solely due to missingness in data for the two variables.

As for the third variable which has missing values — “State” — It has moderately strong relationship between “remote_work”,”tech_company”, “self_employed”, “coworkers” and “comments”.

Here if you see, there is a difference in the chisq test output for — “state” vs “remote_work” & “state” vs “tech_company”, it says that the two variables are independent in complete cases but they end up being dependent in incomplete cases.

In other words, there is something about the missingness in “state” which depends on whether that person is working from home or if he works for a tech_company (aka a fairly strong association in missingness).

Steps further..

Since there is an inherent relationship between the variables that have missing values in our dataset, the data is considered Missing at Random (MAR). Hence, rather than deleting the missing records, it is wiser to do an imputation technique like “mode” imputation, or go for a slightly advanced technique like Multiple Imputation.

That way, we would retain most of the information from our dataset and the accuracy will not be compromised in the end.