Table Design Best Practices for ETL | by Kovid Rathee | Towards Data Science

Not so far ago, the approach taken to table design in source systems (application databases) used to be — we don’t care about ETL. Figure it out, we’ll concentrate on building the application. The last couple of years have been great for the development of ETL methodologies with a lot of open-source tools coming in from some of the big tech companies like Airbnb, LinkedIn, Google, Facebook and so on. And with cloud going mainstream, providers like Azure, Google and Microsoft have made sure that they build upon and support all the open source technologies in the data engineering space.
I have been a part of many ETL projects, some of which have failed miserably and the rest have succeeded. There are many ways an ETL project can go wrong. We’ll talk about one of the most important aspects today — table design in the source system.
ETL pipelines are as good as the source systems they’re built upon.
This statement holds completely true irrespective of the effort one puts in the T layer of the ETL pipeline. The transform layer is usually misunderstood as the layer which fixes everything that is wrong with your application and the data generated by the application. That is absolutely untrue. Without further ago, let’s look at the bare minimum that you should take into account while designing tables which are going to be ETL’d to a target system —
This should have be left unsaid but I have seen systems this is also not enforced (as part of the design). It doesn’t matter if the unique key is a single column or composite in nature. Although, one can potentially be asked to do a full load for a table without a unique key and infer the changes after doing the full load every time. This solution would actually be worse than it sounds.
By enabling data engineers to identify new & updated records by accessing simple fields like created_timestamp and updated_timestamp. Make sure that both these fields are populated by the database and not the application. You should have a separate datetime or timestamp field if you want to populate it from the application. These ones should be defined something like —
1. created_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP2. updated_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
Append only tables with an always-incrementing primary key would work as they are. You don’t need to have audit timestamp columns for those.
In most cases, we’re dealing with relational databases as source systems. It is, therefore, of utmost importance to understand the data flow & lineage within the source system. In most cases, the data flow & lineage in the target system for load remains the same, although that’s not mandatory. Two things will help here — service architecture diagram from the application which produces the data and an ER diagram for the source database.
All these problems can be completely or partially solved even if these points aren’t taken care of in the source system but none of the solutions are going to be sustainable. And yes, that’s all it takes to get started for building a neat ETL pipeline.
As a rule, ETL systems should be tasked just to move data from one place to another with a general layer of transformations (one that doesn’t take care of bugs, special one-off cases). Remember the universal principle of Garbage in Garbage out — if you have buggy data in the source system, your target system will have buggy data.
A practice that I have seen at many places is that data teams try to patch the buggy data and handle-it-for-the-time-being. It usually works until the sprint is over and invariably the same issue comes back to haunt you. Resist the temptation of getting into a habit of fixing issues like that. The right way to do this is to report the issue, fix the data in source system and do a clean reload for the time period in question. A reload will be easier to justify than different data showing up on the application UI and the data BI dashboard.