Joining Data with Version Histories
By: Jonathan Hahn
Tags: Dwh, Data Version Control, Data Warehouse, Data Versioning, Sql

As a data scientist, one often has to struggle with the preparation of data. I am always impressed by the tricks my colleagues in the data warehouse use to solve these challenges using SQL. Recently, I faced the challenge of merging two data sets in SQL, each with its own independent version history, and creating a separate version history. Keeping Track of the old data allows to reproduce a systems data state from the past.

Photo by Tradd Harter on Unsplash

Let’s take a closer look at the problem:

To do this, we reduce the example to the point where an entry in both tables always consists of the 3 values:

ID (for joining the tables)

Valid From (The date from which the entry is valid)

Valid To (The date from which the entry is valid)

As long as the two sources have the same start or end date, this results in only two versions:

Two Versions of the Tables

only values from table 1 are valid: from 20.10.20 to 14.11.20 values from both tables are valid: from 15.11.20 to 03.12.20

With a full join, one could create two entries per ID in the target table. There, the periods could then be adjusted as appropriate. This still needs some SQL skills but with the know how from R or Python one would be able to solve this.

What happens if the time periods differ greatly from each other?

Three Versions of the Table

Now one has 3 different versions in the target table:

only values from table 1 are valid: from 20.10.20 to 14.11.20 values from both tables valid: from 15.11.20 to 03.12.20 only values from table 1 valid: from 03.12.20 to 30.12.20

What follows are the individual steps I took to create my own version history. This is illustrated with a simple example of one ID that is existent in both sources.

Step 1: Create enough rows in the target table

With the union operator, one entry with the start date and one with the end date is created from each table. Thus, for one ID, one gets four entries with the columns ID and date.

+----+-----------+

| ID | DATE FROM |

+----+-----------+

| 1 | 30.12.20 |

| 1 | 03.12.20 |

| 1 | 15.11.20 |

| 1 | 20.10.20 |

+----+-----------+

Step 2: Bring the entries in line

Sort the entries according to the ID and then the descending date. This way one has all entries for the same ID together and the newest date at the top.

+----+-----------+

| ID | DATE FROM |

+----+-----------+

| 1 | 30.12.20 |

| 1 | 03.12.20 |

| 1 | 15.11.20 |

| 1 | 20.10.20 |

+----+-----------+

(This changes nothing in our example, because the data is already sorted. For safety reasons it is still recommended.)

Step 3: Determine time window

To be able to determine the time window, we now have to compare the rows with each other. The LEAD() operator is suitable for this. This is suitable for comparing the current line with the following one.

An entry is then built up as follows:

ID (by which one joins)

Start date (The date from which the entry is valid)

End date (The date from which the entry expires)

The last entry is always set to NULL, because it is not possible to refer to another row from which the entry can be selected.

+----+-----------+----------+

| ID | DATE FROM | DATE TO |

+----+-----------+----------+

| 1 | 30.12.20 | 02.12.20 |

| 1 | 03.12.20 | 14.11.20 |

| 1 | 15.11.20 | 20.10.20 |

| 1 | 20.10.20 | NULL |

+----+-----------+----------+

Step 4: Filter

In the last step, two filters must be set:

Always the fourth row that has the value zero as the end date.

All entries that have the same start and end date.

+----+-----------+----------+

| ID | DATE FROM | DATE TO |

+----+-----------+----------+

| 1 | 30.12.20 | 02.12.20 |

| 1 | 03.12.20 | 14.11.20 |

| 1 | 15.11.20 | 20.10.20 |

+----+-----------+----------+

Here would be a sample SQL Script:

I hope this gives a brief overview of solving the problem that comes with data and versioning. Of course this is only a sample and reality is often way more complex.

BUT FOR NOW … ENJOY CODING!