How To Downfill Null Values In SQL | Towards Data Science

When it comes to data analysis, you often don’t realize what you’re missing until you visualize it. Huge gaps or downward spikes in visualizations will show you exactly where the data is missing, but that’s not a story you want to communicate to your stakeholders.
While some visualization tools can cover these gaps with ease, it’s often best to handle for this at the data source, and relying on your viz tool to do the work for you won’t come in handy if you need to do further analysis outside of tool. So how do we fill in null or missing values at the source?
Let’s suppose we have a table that looks like the one below called inventory_log that keeps track of how many widgets your company has in its inventory at the end of each:
Since your store is closed on the weekends, there would have been no change of inventory from Friday to Monday, but that also means there was no one to count the inventory and to input that value into the table.
It would be safe to assume that the inventory number didn’t change from Friday to Monday, so correct values for Saturday and Sunday would be whatever the value for Friday is. To accomplish this, we apply a tool called filling down. Filling down is exactly what it sounds like: whenever there is a null value, we simply grab the most recent non-null value from above it to replace the null value.
There are many features in Python (including the pandas.DataFrame.ffill() function) that accomplish this, but they are almost always slower than performing the operation directly on the database server.
Now that we’ve fully examined the problem, let’s understand how to do it in SQL.
Using the same table above as our sample data, we can replace the null values utilizing both nested queries and window functions.
The first thing we want to do is to group the rows with null values with the first non-null value above it. We can do that by utilizing a window function to count the inventory column over the date:
select date,       day_of_week,       inventory,       count(inventory) over (order by date) as _grpfrom inventory_log
This query will return us a table that looks like this:
Okay! That gives us a new column that allows us to group together the null values with the first non-null value preceding them. Now the next step is to return that first value for every row that shares a grouping. Luckily, the first_value window function allows us to do just that.
Applying this function to what we already have gives us this query:
with grouped_table as (   select date,          day_of_week,          inventory,          count(inventory) over (order by date) as _grp   from inventory_log)select date,       day_of_week,       inventory,       _grp,       first_value() over (partition by _grp order by date) as filled_inventoryfrom grouped_table
If you want to dive deep into the first_value function, you can check out the documentation here, but the function simply returns the first value for the partition according to the order.
The above query gives us the following table:
That’s it! We now have a column with the null values replaced with the preceding value.
So let’s put our query all together:
with grouped_table as (   select date,          day_of_week,          inventory,          count(inventory) over (order by date) as _grp   from inventory_log), final_table as(   select date,          day_of_week,          inventory,          _grp,          first_value() over (partition by _grp order by date) as              new_inventory    from grouped_table)select date,       day_of_week,       new_inventoryfrom final_table
output:
NOTE: If you have a query where you have multiple dimensions/columns that you need to take into consideration (maybe different store locations in this example), simply add those to the partition clause in the count() window function in the grouped_table sub-query.
Happy coding!
If you want to see other data science tips and tricks, check out my article on How To Load Your Pandas DataFrame To Your Database 10x Faster. Let me know what you’re working on in the comments below!