Building A Modern Batch Data Warehouse Without UPDATEs
By: Daniel Mateus Pires
Tags: Data Warehouse, Apache Spark, Sql, Data Science, Data Engineering

Photo by Startup Stock Photos from Pexels

Managing Dimensions

Static Dimensions

Some special dimensions such as the “date” dimension (a static dimension), are generated, rather than being the result of data processing. Those tables can be generated in advance, and no history of their changes is kept.

When a dimension is static, we can simply overwrite the entire data every time we need to make modifications to it.

Slowly and Rapidly Changing Dimension

For dimensions requiring changes over-time, we need a strategy to mutate the data.

A user can change its home address, a product can change names and a brand can change owners.

Over the years, many “types” of Slowly and rapidly changing dimension management strategies were formalized. Some of them keep history and most of them use “UPDATEs” to add or modify information in-place.

There was a separation between dimensions that have rapidly changing attributes, and ones that have slowly changing attributes.

In a dimension that keeps history, the entire row of data needs to be duplicated when any attribute changes in the record — when the attributes change often, more storage is used.

Those techniques are complex because they were designed under strict storage constraints.

Dimension snapshots use more storage but they are simpler to create and query.

Dimension Snapshots

Dimensions should be much smaller than facts. The transactions, orders, and customer reviews (facts) of an e-commerce might be in the millions, but the number of unique customers (dimension) will be much smaller.

Dimensions also differ from facts due to the “state” they carry. A record in a dimension will have an “identity” that needs to be preserved as other attributes change.

Dimension snapshots are simple to manage

Every day we re-write the entire dimension table in a versioned snapshot

s3://my_data_warehouse/dim_users/ds=2020-01-01

s3://my_data_warehouse/dim_users/ds=2020-01-02

...

Because the entire table is re-written no-matter how many changes happen, there is no advantage in having different strategies for Rapidly and Slowly changing dimensions.

History

With this model, we can answer historical questions easily by joining with a filter on a specific day.

The history is also not slowing down the queries because passing a specific date effectively skips files for any other date.

The structure of the table makes time-series on dimension tables easy. We could, for example, compute the count of users per country of residence over time.

Surrogate Keys

UUIDs introduce randomness. To keep new surrogate keys aligned with the previous snapshot’s surrogate keys we need to query that snapshot.

Conversely, the hash is based on the concatenation of keys that make up the “identity” of a record (excluding the natural key), so we can re-compute it from the current data only. This is why we prefer hashing over UUIDs for dimension snapshots.

Notice that a daily snapshot removes the need for a second key. Some types of slowly changing dimensions implement 2 surrogate keys: one pointing to the “latest” state of the dimension record, and another pointing to the historical snapshot of the record at the time where the specific fact was ingested. Because each snapshot contains all of the data at that point in time, we can use the date of the snapshot to get the “latest” state, or a state at a given point in time.

Won’t that create a lot of data?

Data duplication is the main downside of dimension snapshots.

Another downside is that the dimension needs to be created from one process only, as multiple snapshots a day would aggravate the duplication.

Requirements to refresh dimensions on a more frequent cadence than daily will also amplify duplication.

The gains are simplicity, easy access to history, time-series on dimensions and performance in both writing and querying.

However, for the cases where data duplication is not an option, we can fall-back to another type of Slowly changing dimension management. Using a project like Delta Lake to support UPDATEs.