The Power of Quality Assurance — Designing Robust QA Processes for SQL Data Analysis Pipelines
By: Michael Gendy
Tags: Sql, Pipeline, Qa Testing, Data Science, Data Analysis

SQL Readability and Translation

Often our absolute code efficiency is at the forefront of our coding decision making. Manipulating large volumes of data in complicated queries does require diligent thought to save time and money on processing resource. However, frequently there is the opportunity to trade-off absolute performance with readability to improve how queries are evaluated.

Using a simple SQL example, let’s say we want to list the names of records I own in my record collection that are vintage. We could solve this using the following query:

SELECT t_my.record_name

FROM my_records t_my

INNER JOIN vintage_records t_vin

ON t_my.record_id = t_vin.record_id;

If I owned a very large, entirely impossible collection of billions of records, this is likely close to the best solution, as opposed to using a more computationally heavy sub-query. Yet if I had to hand this over to a friend to run the same query, it doesn’t have that understand-at-a-glance-readability that we might achieve through other means. Consider the questions the new analyst needs to figure out before running the query:

What is the aim of the query?

What are the contents of vintage_records?

Has the INNER JOIN caused any duplicates?

Is record_id a primary key?

In these cases, adding milliseconds to the run time of this query in exchange for some more transferable, readable code is an extremely worthwhile trade-off. By rewriting using a sub query and adding a simple comment, we achieve a much clearer result with very similar performance:

--Returns ALL record names from my collection that are vintage WITH vintage_record_ids AS (SELECT DISTINCT record_id FROM vintage_record) SELECT record_name

FROM my_records

WHERE record_id IN vintage_record_ids;

Same result, similar performance and much improved readability. Notice:

Clear purpose allowing for quick interpretation Use of the WITH statement allowing readability from the top-down Clear name to the sub-query table outlining it’s contents Eliminating the need for a join simplifying the code

Good readability best-practices include commenting out the purpose of the query, utilising WITH statements (CTEs), capitalising functions, keeping queries short and having clear processing logic. Thanks to our changes, we now have clarity on many of the query’s nuances. We now know it’s purpose and that we’re returning all results from my record collection (not just unique results). We can interpret the code faster thanks to the sub-query, and don’t have to worry about any faulty JOIN logic. Slower but cleaner code is definitely the winner here.

While it may seem like it saves a small amount of time on a simple query like this, the actual value it derives will continue to pay off every time this is run. When your SQL infrastructure involves complex, multiple thousand line scripts all integrated with one another, the pay-off for writing your code in a clear manner increases exponentially. Not to mention the often overlooked factor in smaller teams — what happens when the primary analyst for these scripts is on holiday or off sick? Can another analyst pick up the script effectively? An analyst contributes greatly to making business operations more efficient, and so it’s crucial to apply this to their own processes and practices too, in the form of producing readable and transferable code.

Unit and Integration Testing

Testing and retesting any production script is the biggest non-negotiable of all. We must both test code in isolated segments (unit testing), and test how well these isolated segments work together (integration testing). For example, testing that a temporary table is created correctly, then ensuring that any queries that use the temporary table interact with it correctly.

SQL is deceivingly simple, and learning to anticipate duplication and becoming accustomed to diagnosing faulty joins can bring a lot more peace to your day-to-day analysis. Commenting out code to evaluate if each query has produced the intended result (unit testing), and making sure they won’t break any of your other dependent queries (integration testing) is a skill worth developing in itself. At a minimum, you should include unit testing queries and comments which answer the following questions:

Are my counts before and after JOINs expected?

Have any of my primary keys been duplicated?

Are my JOIN fields in the same format?

Are my CASE WHEN statements working as intended?

Does the result make logical sense?

The next step is ensuring these individual queries work in harmony with any other queries in your script. Checking that one section of code is working correctly but failing to test how this query fits into your larger data pipeline can create unnecessary and easily avoided vulnerabilities.

Take the time and document your thought process — this will pay off greatly by preventing errors and ensuring bugs are identified and solved swiftly. Give as much detail as you can in a clear, succinct manner. If the result is correct, but counter intuitive — explain it in a comment! The goal here is to make your outputs reproducible and allow others to easily verify the result. Outlining the above during SQL script writing will soon become second nature, and you’ll develop your own intuition for bugs that arise repeatedly. In addition to building more confidence in our work, we also give our peers the ability to pick up the script, understand the purpose and evaluate its efficacy with minimal hassle.

Monitoring Your Data Flows

It’s genuinely quite incredible how many seemingly insignificant changes to your data ingestion can affect your data pipelines. Here’s a handful of examples:

NULLS appearing due to trafficking issues

Formats of dates, internal IDs, country codes changing between suppliers

Duplicates appearing due to API glitches

Manually input data being incorrectly filled in

Any of these issues can affect multiple queries in seemly obscure ways, from causing JOINS to fail, breaking NOT IN statements and messing with your WHERE clauses. Without clear visibility on your data at every processing stage, often these issues can go unnoticed causing a headache down the line. Odds are, the longer the issue has been present, the bigger the consequences — catching any of these issues early can be the key to smooth operation.

Clear data visualisation is the key to solving this. An easy way to do this is by creating a pipeline monitoring table, inserting counts of key metrics at every stage of the data processing, and visualising the outputs in a BI tool:

INSERT INTO qa_pipeline

(SELECT table_name

,CURRENT_TIMESTAMP() as run_timestamp

,count(*) as row_count

,COUNT(primary_key) as primary_key_count

,COUNT(distinct_primary_key) as distinct_primary_key_count

,COUNT(*) FILTER (WHERE important_field > '') as important_field_population

,SUM(important_kpi) as important_kpi_sum

FROM raw_table)

This is a very useful, general format of how we want to track our data. Creating visualisations of a table such as qa_pipeline, and tracking important values such as row counts, field population, KPI sums and primary key interactions between queries can enable at-a-glance debugging. For example, in your raw tables, your primary key will always be unique. In your cleaning script where you create a new table and join in extra information, if we now see the count of primary keys increase, we know there’s some duplication caused by the cleaning script. Inversely, if we see the distinct primary key count decrease, we know some have been filtered out:

Example visualisation of a complete data pipeline on the 1st January 2020 (in Qliksense)

The visualisation above demonstrates the power of these qa_pipeline tables. It’s immediately obvious that every metric doubles in the final table, indicating that there’s faulty logic in the script to produce the final_output which is causing duplication. A number of issues can be discovered by simply getting a pair of eyes on the true outputs of each script or query. If we saw a significant decrease in important_kpi_sum, this would suggest that there’s an error when using important_kpi or a mistake in the aggregation logic — a clear sign that the code should be reviewed. While manually going through entire outputs regularly can demand too much resource for many analytics teams, a quick inspection of these summary visualisations every morning will lead to more proactive solutions to diagnose incorrect SQL scripts.

Another use case of the same table is making sure your data is being ingested correctly — the quality of your raw data will mirror the quality of your outputs. Viewing your ingestion table metrics using the qa_pipeline table visualisations allows for easy monitoring that your data is coming in the intended formats. Anomalous results such as low row counts, missing days and high numbers of missing fields can be identified easily and quickly:

Example visualisation of the raw_table population over the first week of January (in Qliksense)

At a quick glance, we can see there are some ingestion issues for our raw_table. On the 3rd of January when the script was run, we saw three times the amount of rows as we see distinct primary keys, indicating that the data may have been ingested three times. In addition, on the 5th of January, we can see our important_field was not completely populated, hinting at some tracking issues. After only a few seconds of looking at the visualisation, we can clearly evaluate the quality of the data and highlight any issues, further improving our ability to react to problems and resolve them efficiently.

Use your creativity in how you display these counts to suit your business use case. Your intuition is a valuable resource here — learn to anticipate how your code will affect your data and compare this consistently to the visualisations provided by these qa_pipeline tables. Setting up these INSERT INTO queries at ingestion, during data cleaning and after the deliverables have been produced gives you end-to-end clarity on how your scripts are performing, saving valuable resources on debugging and preventing any errors.

Reducing Dependencies

A competent analyst can work through any problem. In theory, all code should be running perfectly if the quality of the data remains the same. However, when managing the ingestion of multiple data streams from different sources, this data quality is never guaranteed. Possibly the simplest way to maintain good quality data is by taking control of it yourself, allowing full autonomy and greater reactivity when problems arise. Next time you’re ingesting third party data, consider the following:

If we found an error, how quickly could it be solved by the source? Could I gain autonomy of this data by producing it in-house? Could any manual processes be automated to reduce human error? How can we work with the sources to reduce our reliance on them?

Consider this scenario:

“An international third party is supplying time series data in the form of a daily excel report. Due to daylight savings in their region, the timestamps they have provided are all an hour out, which means you’re missing an hour of data in the report provided. They’re in a completely different timezone, won’t be able to rectify the issue until the morning, and the clients demanding a crucial report be delivered by the end of the day.”

Not a fun spot to be in. You could make a strong case that the third party is not even complicit, because the process is inherently flawed for many reasons. However, if we had autonomy over the data they provided, by querying their API directly for example; we could simply make another call to correct the time frame within minutes. No more missed deadlines and no more unhappy clients.

In reality, third parties may not have the same resources, infrastructure, practices or approach that you may be used to. This difference can lead to errors in ingestion, processing or presentation which may be slow to fix if you need to ask a third party to reformat or reproduce the data for any reason. Retaining as much autonomy over your data as possible enables you to be more reactive to changes, find solutions faster and optimise the production to a level that you control.

Peer Review and Optimisation

A key attribute that’s helped me throughout my time as an analyst is my ability to learn from others. I really enjoy learning why people are successful at what they do and the traits they possess which I can apply to my own life and career. Consistent, constructive feedback from others can really accelerate your learning and challenge you to perform better, and peer review is crucial to this in an analysis environment. Having a colleague review your code can have many benefits:

Adding further confidence to your data outputs

Creates an opportunity to learn new techniques

Shares knowledge between teams and peers

Provides another angle of thinking which provokes discussion on methodologies and promotes more innovative solutions

A fresh pair of eyes can be incredibly impactful to improving your code. Having someone approach the same problem, think through the same process and come to the same conclusion is so beneficial for someone that’s been staring at the same multi-join for the last 2 hours, not able to figure out if it’s working properly. I can’t remember how many times I’ve walked over to a teammates desk with a bug I’ve been stuck on, only for them to spot it within 30 seconds, or suggest a much more efficient way of achieving the desired result. One of the biggest reasons for this is that often bugs or duplicates appear through the most difficult to notice, obscure and inconspicuous SQL interactions involving the nichest edge cases.

As an analyst, your ability to identify potential edge cases and account for them, ideally before they even occur, is paramount to preventing any problems before they even materialise. Thinking of every potential edge case alone is extremely difficult and nigh on impossible, so utilise the best resource you have — productive discussion with your team and peers. Building consistent peer review into your workflow, to ensure that no code goes to production without being seen by at least one colleague, goes a long way to increasing your own confidence as an analyst, and can drastically speed up delivery of work.