BigTips: Removing Duplicates while Maintaining Row History
By: Brian Suk
Tags: Google Cloud Platform, Analytics, Bigquery, Data Warehouse, Big Data

When working with and implementing BigQuery, there’s a number of small problems to which I couldn’t find documentation to, or a working example of a solution. This occasionally happens with any database. While these won’t probably be groundbreaking problems to solve, hopefully it’ll make someone’s day a little easier. Sometimes, it’s the little things.

BigTips: Removing duplicate rows with mixed static and variable columns, while keeping row version histories!

I Know What I Need, Just Point Me To The Scripts!

The Problem Statement

One of the age old problems in analytics data is how to deal with late arriving data. This is a fairly common problem when ingesting data, especially with systems that deal with frequent transactions. It’s also an issue with systems that often times issue corrections to previous data.

The challenge statement here is trying to handle this issue within BigQuery itself. ETL tools like Informatica PowerCenter, IBM DataStage, and their usual peers all have facilities to handle this within their tool. While this is usually fine when running your ETL pipeline, what if you wanted to leverage BigQuery’s engine to do it?

A couple ideas initially come to mind. The first, and most obvious one, is to just issue a SELECT DISTINCT * statement to find and handle all the duplicate rows. If all you’re looking for is that, then that could work. What this does not handle are updates to previous entries, as well as updates when some columns are dimensions, and others are measurements. It would treat an update as a new distinct row, so you wouldn’t be able to easily tie those together.

Another method is to use the MERGE statement. This has applicability to the problem statement, but it’s not entirely clear how we would handle a variable number of surrogate key columns, and how to exclude different ones. Especially if we wanted to make this generic and apply to an unknown table.

These objections might be confusing when describing it here in text descriptions, but let’s give an example to make this concrete.

Let’s say we have a base table called main_table with the following base data:

Here are what the columns mean:

is_latest is a BOOL column which tells us if that row represents the “current” version of that data.

is a column which tells us if that row represents the “current” version of that data. ingest_ts is a column which tells us when that row was loaded into the table.

is a column which tells us when that row was loaded into the table. reported_dt is a column which tells us what date the data in that row represents.

is a column which tells us what date the data in that row represents. COLA is just some dimension column, in this case it’s a state.

is just some dimension column, in this case it’s a state. COLB is another dimension column, in this case it’s an age group.

is another dimension column, in this case it’s an age group. MEASUREMENT is just some measurement of something.

is just some measurement of something. COLC is another measurement column. This one doesn’t have any particular meaning to the data, it’s just there as another column where the value can change.

Here we have both ingest_ts and reported_dt so that the data is bi-temporal and we can measure data both “as it is when the source observed it” and “as it was when we received it.” The idea here is to keep a running history history of changes (by using a combination of both ingest_ts and reported_dt ) as well as maintaining the is_latest flag so you can easily just issue the query SELECT * FROM mytable WHERE is_latest = TRUE to get all the latest data.

In this structure, we have three groups of columns.

is_latest and ingest_ts are metadata columns which help us calculate recency of the data, but in itself isn’t the actual data. is_latest is a simple filtering flag that needs to be maintained.

and are metadata columns which help us calculate recency of the data, but in itself isn’t the actual data. is a simple filtering flag that needs to be maintained. reported_dt , COLA , and COLB are the three columns which as a combination can tell us what represents a unique measurement of data. These can act as our surrogate keys. A unique combination of these columns means it’s a “new row” in a sense. Throughout this we will refer to these as the “unique columns.”

, , and are the three columns which as a combination can tell us what represents a unique measurement of data. These can act as our surrogate keys. A unique combination of these columns means it’s a “new row” in a sense. Throughout this we will refer to these as the “unique columns.” MEASUREMENT and COLC are our measurement columns. For every unique combination of the surrogate keys, it’s possible for these two measurements to change. The next dataset will give an example of how that’s possible. Throughout this we will refer to these as the “measurement columns.”

So now that we have the base data table, let’s add a second table. This is our ingest_table which has a similar structure, but this represents new data that has arrived on “a later date.”

This is just a subsequent data load of the same table, so the is_latest column doesn’t exist there (that’s what we need to calculate), and here’s some examples of data intricacies.

California has an update (for 2020–1–15, California’s 18–22 count goes from 150 to 200), a new row (for 2020–1–15, California’s 22–30 is a new record), and a duplicate from history (for 2020–1–15, California’s 18–22 measurement of 100 and ‘hello’ is a historical duplicate).

Delaware’s row (for 2020–1–15, Delaware’s 22–30 count of 100/pong) is a duplicate of an active data point.

Florida’s measurement (for 2020–1–15, Delaware’s 22–30 count of 200/ping) is a new row.

This is why strategies of just matching on a column doesn’t always work, nor does just simply selecting distinct across all columns. It’s a little trickier. Also, this does happen in real world systems a lot. If you’re a retailer, it could be a scenario of, “our production line had issues and last Tuesday’s batch actually has 100 less units than we originally recorded.” Or if you’re aggregating healthcare data nationally, you might have to update data in a situation of, “the measurement for three days ago is actually different because some hospitals in California got held up with paperwork, and we just now got that data in.” More tangibly in your every day life, think of your credit card statements. If you watch it frequently, you’ll notice there’s usually a lag period of a few business days as transactions are cleared (at least here in the United States, but I suspect many countries operate similarly), and your “as of” balance and your “current balance” during those days may fluctuate as transactions close out over time. These are all examples of when this issue comes into play.

Let’s go back to our example tables. Embedded in there are examples of the most common late arriving data scenarios: Truly new unique data, updates to the latest data, duplicates of the latest data, and duplicates of previous historical data. It’s pretty much an UPSERT . Let’s go and build something that handles that!

We’re going to walk through how to build this piece by piece, and will also wrap this around a stored procedure. If you just want to skip to the final result, go here for the example with data embedded into it, and go here for the actual script you can deploy.

Just so we know what our target end state is, if we apply the data changes from the ingest table to the main table manually we can see that this is what we want:

Let’s try and get there.

We’re going to take two approaches with this. A fully self contained BigQuery Script that includes sample data, and also a version where we wrap that all up in a stored procedure so you can call it on arbitrary tables.

The Psudeocode

Let’s get our logic down first, so we know what we’re building towards. The basic psudeocode for this looks like the following:

Let’s see how this logic handles the four data scenarios outlined earlier:

Updates to the latest data: The first half of the WHERE clause in Block 1 addresses this. It looks for rows in main_table where is_latest is true, the unique columns match signifying it’s the same measurement, but with new measurements. Block 2 is where we then mark the corresponding in main_table as a historical one, and then load the new data in Block 3.

clause in Block 1 addresses this. It looks for rows in where is true, the unique columns match signifying it’s the same measurement, but with new measurements. Block 2 is where we then mark the corresponding in as a historical one, and then load the new data in Block 3. Duplicates of previous historical data: The catch with the previous logic is that if an incoming row has previous historical measurements, it still could pass the logic since it’s not comparing it to historical values. The second clause in Block 1 is where we catch this exception, and ignore it when creating our staging table of updates.

Duplicates of the latest data/Truly new unique data: Block 4 handles both of these. Here we only merge in rows where both the unique and measurement columns all are new.

Comparing with JSON

When doing these comparisons to see which columns have changed and which haven’t, we would normally have to manually type all these out in a where clause. Not only is that a pain with very wide tables, but it also isn’t flexible to work generically on tables. The idea of working on generic tables bit we’ll address in a later section, but let’s bring in JSON strings to address the other issue.

One of the tricks we can do to compare a large number of columns in a simple statement, instead of having to do a bunch of comparison operators, is to just serialize the row as a JSON string and do one comparison. You can use the TO_JSON_STRING() function to easily serialize the JSON strings.