How to Deal with NULL Values in Standard SQL
By: Marie Lefevre
Tags: Programming, Bigquery, Database, Sql, Data Science

Step #1: Identify & Visualize

First, we want to identify if there are any NULL values in our dataset. There are two ways to look for null values in a dataset, depending on your prior knowledge of the data you are manipulating.

If you already know in which field (or column) there may be NULL values that you want to target, you can directly apply the condition WHERE IS NULL on the field of interest. In our example, if we know that ‘country’ is not a mandatory field while we want to analyze the split of our customers by geographical area, we would like to start investigating this column only. This is possible with this piece of code:

If you don’t have a precise idea of the fields you are investigating — or if you want to get a complete overview of null values in your dataset, you must start by getting a clear idea of the possible existence and the number of NULL values per column. To get this overview, you can use the following code snippet (retrieved from Stack Overflow):

In our case, the output is the following:

© Marie Lefevre

Step #2: Understand & Decide

Now that you identified where null values are present in your database, you can go one step further and investigate what brought them here. There are many reasons why a database contains null values. Let’s answer three “background check” questions to better understand them.

1. Are all rows NULL for some fields?

In our example, the field ‘age’ is always empty. One reason for this could be that the field is not mandatory for the website’s customers: when ordering online, you don’t need to communicate your age except for specific items. As age is personal data, clients may not be keen to share this information when doing online shopping.

This can also indicate a potential bug on the website. In both cases, it is critical for any business to monitor the number of occurrences of null values in a database — and how it evolves over time — especially when all entries are null. In this case, this could lead to a more profound reflection with the product team: is it even relevant to keep asking this information to customers if nobody fills it in?

2. Is the same number of rows NULL for several fields?

Intentionally or not, some fields can be related to each other, the information filled in one of them depending on the other. Therefore better understanding the inter-dependencies of fields can help you interpret why some fields have a given number of null values.

In our example, this would correspond to the fields ‘state’ and ‘us_taxpayer’: these two columns have 7 null values each. When looking at the whole table, we see here that customers who did not complete the field ‘state’ live in countries other than the United States. If the ‘us_taxpayer’ field is not mandatory, this would explain that these seven customers did not need to check the last box when ordering online “Are you a US taxpayer?”.

3. What are the remaining cases?

Ultimately you need to look at the edge cases that remain unexplained after the previous two steps. They can correspond to errors in data entry or simply to outlier cases. In both cases, investigating what led to these remaining null values can be very instructive to improve your product or your user experience for instance.

In our e-commerce example, it is intriguing that an online customer (Carla C.) did not provide their email address. How is it possible that this field is not mandatory although it is necessary to receive the order confirmation and the purchase bill via email? In any case, this kind of data entry must be made impossible in the future. This is why investigating what caused this null value is crucial for business: it may be due to a human error (e.g. the client forgot to give their email address) or a technical error (e.g. the website encountered a bug when saving the form, hence the value filled in could not be saved).

Photo by Hadija Saidi on Unsplash

Step #3: Act & Modify

Depending on the conclusions you drew from the previous steps, it is now time to have an impact on these null values. Should they remain like this or should they be removed at all costs? I identified three ways to handle them:

replace the null value by a neutral value, e.g. “N/A” or “-”

replace the null value by a meaningful value

let the null value be null

For the cases when you decide to replace the null values, let’s apply the most appropriate SQL functions.

IFNULL: the simplistic one

As its name suggests, the function IFNULL(expr, null_result) returns a certain value if the input value is null.

In our example,we want to make null values from the field ‘state’ more explicit so that we choose to replace the null values by a neutral term like “data not available”. The same applied for the field ‘age’. In addition we want to remove all null values from the field ‘us_taxpayer’. Let’s assume that all customers who did not answer “yes” to the question are indeed not paying taxes in the United States. Therefore, if this field is null, we want to return “no”. Combined with the previous change, this would give the following code snippet:

IFNULL(state, ‘data not available’)

IFNULL(age, ‘data not available’)

IFNULL(us_taxpayer, ‘no’)

COALESCE: the listing one

The function COALESCE(expr[, ...]) returns the value of the first non-null expression. To use it, you will need to define the priority order of the terms you want to use to replace null values beforehand.

In our case, we want to ensure that the field ‘email’ has no null value. A reason for this could be that this field is used by our support team as the contact point to customers. So we want that this field equals, in the following priority order:

If available, the customer’s email address Else, if available, the customer’s phone number Else, the customer’s name

To do so, we cannot use the IFNULL function as it can only display one alternative value for null values. So we use the following piece of code:

COALESCE(email, phone, name)

CASE WHEN: the complex one

Finally the CASE expr WHEN expr_to_match THEN result [...] [ELSE else_result] END function can be used when several cases need to be handled, including more complex cases where IFNULL or COALESCE are not sufficient to replace null values.

In our example, in cases where the field ‘country’ is null, we want to apply the following rules in a specific order:

If ‘country’ is not null, then let’s take this value Else, if ‘state’ is not null and belongs to a state of the United States, we want to write “United States” in the field ‘country’ Else, if ‘state’ is not null, we want to replicate this ‘state’ value into the ‘country’ field Else, let’s write “no country”

In standard SQL, this corresponds to the following piece of code: