SQL Unit Testing in BigQuery? Here is a tutorial
By: Mike Shakhomirov
Tags: Analytics, Unit Testing, Editors Pick, Big Data, Data Science

Use case

Our test will be a stored procedure and will test the execution of a big SQL statement which consists of two parts:

First part generates a source dataset to work with. Here we will need to test that data was generated correctly.

The second one will test the logic behind the user-defined function (UDF) that will be later applied to a source dataset to transform it.

All the datasets are included. They are just a few records and it won’t cost you anything to run it in BigQuery.

Who is this article for?

It is for everyone who would like to:

Learn how to write scripts in BigQuery. Learn about SQL unit tests. Make data more reliable and/or improve their SQL testing skills.

Why do we need unit tests in SQL?

I have run into a problem where we keep having complex SQL queries go out with errors.

Chaining SQL statements and missing data always was a problem for me.

That’s why, it is good to have SQL unit tests in BigQuery so that they can not only save time but also help to standardize our overall datawarehouse development and testing strategy contributing to streamlining database lifecycle management process.

Unit tests in BigQuery (standard) SQL

A typical SQL unit testing scenario is as follows:

Create BigQuery object (dataset, table, UDF) to meet some business requirement. Create a SQL unit test to check the object. Run SQL unit test to check the object does the job or not. If the test is passed then move on to the next SQL unit test.

During this process you’d usually decompose those long functions into smaller functions, each with a single clearly defined responsibility and test them in isolation.

I would do the same with long SQL queries, break down into smaller ones because each view adds only one transformation, each can be independently tested to find errors, and the tests are simple.

So let’s test the first part of our long SQL statement.

Test 1.

Let’s imagine we have some base table which we need to test. For this example I will use a sample with user transactions. This is a very common case for many mobile applications where users can make in-app purchases, for example, subscriptions and they may or may not expire in the future.

Run this SQL below for testData1 to see this table example. It’s a CTE and it contains information, e.g. user_id , product_id , transaction_id , created_at (a timestamp when this transaction was created) and expire_time_after_purchase which is a timestamp expiration for that subscription.

Test table testData1 will imitate a real-life scenario from our resulting table which represents a list of in-app purchases for a mobile application.

I want to be sure that this base table doesn’t have duplicates. For example, ‘For every ( transaction_id ) there is one and only one ( created_at ):

Here’s our first test:

Test 2.

Now let’s test it’s consecutive, e.g. our base table is sorted in the way we need it. Even though BigQuery works with sets and doesn’t use internal sorting we can ensure that our table is sorted, e.g. consequtive numbers of transactions are in order with created_at timestmaps:

Now let’s wrap these two tests together with UNION ALL: