Building A Modern Batch Data Warehouse Without UPDATEs | by Daniel Mateus Pires | Towards Data Science

In this post, I will describe how to design Dimensions, Facts and the processes that feed them without performing mutable changes.
For this post’s purpose, I will assume the data is on a cheap cloud storage (S3 / GCS), processed by a compute engine that operates directly on the files (Spark / Hive), and written in immutable blocks (Parquet / Avro).
Know that there are options for transactionality, updates and locking in a Big Data stack. Databricks and Snowflake built very successful businesses around providing those services (and more). Databricks even open-sourced “Delta Lake”, bringing some of those features to Apache Spark.
However, UPDATEs are complex operations in a Big Data stack. And I believe that keeping most of our data processing immutable is simpler to maintain and reason about.
UPDATEs introduce a, perhaps, unwanted flexibility to a data pipeline. I have seen a team extend tables owned by another team only through (highly inefficient) UPDATEs instead of collaborating and extending common scripts. Immutability forces a data pipeline to have a certain structure.
I will start by describing, at a high level, Star Schema concepts, and explain why they are still relevant today. If you are familiar with those concepts, I invite you to skip ahead to “Managing Dimensions”.
The Star Schema is a type of relational modeling technique that separates measures or events (facts) relating to a business process from their context (dimensions).
It is more denormalized than relational database schemas (application databases), and was designed to be efficient for Analytic queries (OLAP workloads).
Example:
In an e-commerce website, an example of a business process is a transaction on the website.
Some example measures, in this scenario, are: the amount paid during a transaction and the quantity of product bought.
The context is made of: the user that made the transaction, the date of that transaction and the product bought.
In reality, each transaction could contain different products: this is a simplified model
The star schema gets its name from the physical model’s resemblance to a star shape with a fact table at its center and the dimension tables surrounding it representing the star’s points
https://en.wikipedia.org/wiki/Star_schema
The dimensions can be further normalized: for example the “brands” of products could be kept in a separate table with a foreign key relationship to “products”, creating a dimension to dimension relationship.
The further normalization of the dimensions of a Star Schema results in a “Snowflake Schema” and, historically, its main objective was to reduce the amount of redundant data (save storage).
The downside is that a Snowflake Schema introduces more complexity through more joins. As storage became less of a concern, a “Snowflake Schema” approach is not appropriate for the majority of cases.
The Star Schema was created in a time where storage and compute were expensive. Because storage was expensive and limited, reducing data redundancy was a main concern of the Data Warehousing team.
It was also an efficient way to support Data Warehouse queries as large amounts of data could be skipped on fact tables through JOINs and filters on dimension tables. The predictable access patterns allowed for simple optimizations, such as the creation of indexes on the foreign keys of fact tables.
Short and sweet — all foreign key columns should have a non-clustered non-unique index.
https://www.datavail.com/blog/how-to-index-a-fact-table-a-best-practice/
Today, storage is cheap and it is (relatively) easy to provision compute power as needed. We could measure the engineering cost of designing and implementing those models against the amount saved on hardware, and ask ourselves — is it still worth doing?
Companies collect more and more data from an increasing number of sources, and the resulting datasets need to be reconciled for analytics.
For example, a cable network might have very different systems to host the information about their TV subscribers and the subscribers of their newly launched streaming services. Similarly, their “customer support” analytics has to integrate data from Twitter, their 3rd party call center and their support emails.
The normalization effort has the objective of bringing together the different definitions of “subscribers”, “customer feedback” and other logical entities, removing the analytics complexity that would be otherwise introduced by the heterogeneous source systems.
The Data Warehouse Toolkit (1996) written by Ralph Kimball, and the Kimball Group website define concepts widely understood in the industry (such as the Star Schema).
New hires can quickly get a grasp of the Data Warehouse structure without being familiar with the specifics of the organization.
Data engineers, data scientists and analysts have common terminology (fact, dimension, grain), facilitating collaboration.
A newly added fact can re-use the existing dimensions.
New dimensions can be added to facts by adding more foreign keys to the fact tables.
As a result, new datasets can be integrated without introducing major changes to the schema.
A Star Schema can be populated purely through inserts and computations easily parallelize-able within a Map-Reduce framework (we will see “how” in the next sections).
Query performance tuning:
Although foreign key indexes are typically not an option because they are not supported in many modern Data Warehousing frameworks — we have other options to increase performance.
Dimensions are typically small, and can sometimes fit in memory, enabling Map-side join optimizations (see Spark’s BroadcastJoin).
We can use bucketing on the dimension tables.
Spark has support for Star Schema detection, to re-order JOINs and efficiently skip data.
Many of the practices from 1996 are still relevant today, but some methodologies need to be redefined.
Companies such as Lyft have been successful at updating those Data Warehousing practices to adapt to the new technology landscape. Maxime Beauchemin has a great talk on the subject, and the “Managing Dimensions” section of this post is heavily inspired from it.
From the same talk:
First learn the rules, then break them
The Data Warehousing books warned against re-using “natural keys” (unique IDs from the production systems) as foreign keys in fact tables.
Natural keys in the production systems could potentially re-use IDs if a table is partially emptied (for performance reasons). Because the Data Warehouse keeps historical data, the re-use of IDs creates clashes complicated to resolve.
The best practice for the creation of “surrogate keys” was to use integer IDs sequentially generated by the data processing system, and detached from the production systems’ natural keys.
Integers allowed saving storage and creating smaller and efficient indexes.
Indexes are not used in modern data warehouses. Hive 3.0 removed indexes, they are replaced by statistics / metadata compiled in the files (ORC, Parquet), physical partitions and bucketing which, similarly, enable skipping large portions of data.
Generating surrogate keys is a complex operation to parallelize.
This BigQuery (Google) blog post describes the limits of adding a generated sequence using the common approach (ROW_NUMBER function over the new rows):
Unfortunately, this approach is limited. To implement ROW_NUMBER(), BigQuery needs to sort values at the root node of the execution tree, which is limited by the amount of memory in one execution node.
UUIDs and hashes are easier to parallelize, making it a more scalable approach to generating IDs for “Big” datasets.
For “dimension snapshots” we will prefer hashes over UUIDs, we go through the motivations in the below sub-section: “Managing Dimensions” / “Surrogate Keys”.
Some special dimensions such as the “date” dimension (a static dimension), are generated, rather than being the result of data processing. Those tables can be generated in advance, and no history of their changes is kept.
When a dimension is static, we can simply overwrite the entire data every time we need to make modifications to it.
For dimensions requiring changes over-time, we need a strategy to mutate the data.
A user can change its home address, a product can change names and a brand can change owners.
Over the years, many “types” of Slowly and rapidly changing dimension management strategies were formalized. Some of them keep history and most of them use “UPDATEs” to add or modify information in-place.
There was a separation between dimensions that have rapidly changing attributes, and ones that have slowly changing attributes.
In a dimension that keeps history, the entire row of data needs to be duplicated when any attribute changes in the record — when the attributes change often, more storage is used.
Those techniques are complex because they were designed under strict storage constraints.
Dimension snapshots use more storage but they are simpler to create and query.
Dimensions should be much smaller than facts. The transactions, orders, and customer reviews (facts) of an e-commerce might be in the millions, but the number of unique customers (dimension) will be much smaller.
Dimensions also differ from facts due to the “state” they carry. A record in a dimension will have an “identity” that needs to be preserved as other attributes change.
Dimension snapshots are simple to manage
Every day we re-write the entire dimension table in a versioned snapshot
s3://my_data_warehouse/dim_users/ds=2020-01-01s3://my_data_warehouse/dim_users/ds=2020-01-02...
Because the entire table is re-written no-matter how many changes happen, there is no advantage in having different strategies for Rapidly and Slowly changing dimensions.
History
With this model, we can answer historical questions easily by joining with a filter on a specific day.
The history is also not slowing down the queries because passing a specific date effectively skips files for any other date.
The structure of the table makes time-series on dimension tables easy. We could, for example, compute the count of users per country of residence over time.
Surrogate Keys
UUIDs introduce randomness. To keep new surrogate keys aligned with the previous snapshot’s surrogate keys we need to query that snapshot.
Conversely, the hash is based on the concatenation of keys that make up the “identity” of a record (excluding the natural key), so we can re-compute it from the current data only. This is why we prefer hashing over UUIDs for dimension snapshots.
Notice that a daily snapshot removes the need for a second key. Some types of slowly changing dimensions implement 2 surrogate keys: one pointing to the “latest” state of the dimension record, and another pointing to the historical snapshot of the record at the time where the specific fact was ingested. Because each snapshot contains all of the data at that point in time, we can use the date of the snapshot to get the “latest” state, or a state at a given point in time.
Won’t that create a lot of data?
Data duplication is the main downside of dimension snapshots.
Another downside is that the dimension needs to be created from one process only, as multiple snapshots a day would aggravate the duplication.
Requirements to refresh dimensions on a more frequent cadence than daily will also amplify duplication.
The gains are simplicity, easy access to history, time-series on dimensions and performance in both writing and querying.
However, for the cases where data duplication is not an option, we can fall-back to another type of Slowly changing dimension management. Using a project like Delta Lake to support UPDATEs.
Fact tables are the foundation of the data warehouse. They contain the fundamental measurements of the enterprise, and they are the ultimate target of most data warehouse queries
https://www.kimballgroup.com/2008/11/fact-tables/
There are multiple types of fact models to cover the different measures and events that a fact table aims to capture.
When designing our fact, we need to decide the level of detail of each row in the table.
Pre-aggregating facts could save storage, but it is a risky bet to make, as analytic requirements from the business can require more details in the future. For most cases today, capturing fact information at the lowest level of detail is desirable.
The vast majority of Facts should not require UPDATEs. Even in the specific case of Accumulating Snapshots, we can design the model in a way that removes the need for mutability.
For an e-commerce order for example, if we have the states: ORDER_BEING_PREPARED, ORDER_SHIPPED, ORDER_IN_TRANSIT. Instead of going to the fact table and updating the row with an end_date every time the action finishes, we can create new states that symbolize the end of previous states: ORDER_BEING_PREPARED_END, ORDER_SHIPPED_END, ORDER_IN_TRANSIT_END. Now each change of state is a separate event that can be inserted.
In order to create immutable partitions that can be overwritten in case of upstream errors or changes in business requirements, we need to partition our data by the “extraction date” — the date when the data was extracted from the source system.
This first partitioning key is typically not useful for query performance, but it is important in order to implement idempotent batch processing jobs.
Additionally, partitioning by the “event date” is recommended as queries will often use it as a filter (WHERE clauses).
Other partitioning keys can be added to additionally increase query speed, but we need to make sure they are used in filters, and the resulting files are not too small.
Example table structure:
under s3://my_data_warehouse/fact_transactions:/event_date=2020-02-05/extraction_date=2020-02-05/event_date=2020-02-05/extraction_date=2020-02-06/event_date=2020-02-06/extraction_date=2020-02-06...
Our data processing job created 1 partition on the 5th: the /event_date=2020–02–05/extraction_date=2020–02–05 partition.
On the 6th however: some transactions that took place on the previous day were captured late, so 2 partitions were created: /event_date=2020–02–05/extraction_date=2020–02–06 and /event_date=2020–02–06/extraction_date=2020–02–06 , this is commonly known as “Late arriving facts”, and is supported by this table structure.
We can avoid UPDATEs, locking and the need for ACID compliance in a Data Warehouse by adapting the “Star Schema” approach to leverage cheap Cloud storage.
Foreign keys are hashes instead of integer sequences
Dimensions use daily snapshots
Facts are partitioned by “extraction date” to support idempotent processing and the potential overwriting of partitions