Explore Arrays and Structs for Better Query Performance in Google BigQuery
By: Skye Tran
Tags: Data Warehouse, Data, Sql, Bigquery, Tutorial

Arrays

Definition

An array is an ordered list containing values of the same data type. It is used when we want to store repeated values in a single row such as those below scenarios.

An employee has one or multiple skills

A student can speak one or multiple languages

A restaurant listed on Zomato has one or multiple cuisine labels such as Italian, Pizza, Pasta, Casual

A customer’s sales order includes one or multiple items

Working with SQL Arrays

Create an array with square brackets

Let’s begin with something simple. We will attempt to create a permanent table with an array to capture cuisine labels associated with 4 different restaurants in Sydney.

Photo by author

Looking at the query result, here are 3 key points to note.

We can create an array with square brackets [ ] with each element separated by a comma. Values within each pair of square brackets form 1 array. Since each array represents cuisine labels associated with a restaurant, BigQuery has stored them in repeated fields associated with one single row. In total, we have 4 different rows corresponding to 4 restaurants. Under each row, cuisine labels are displayed vertically in different lines instead of lumping them together inside square brackets horizontally. We say BigQuery has naturally flattened the output.

Break open an array with UNNEST

What if you want to unpack the array so that it can be exported to a spreadsheet in MS Excel or Google Sheets? Well, fair enough, UNNEST operator will come in handy for such a case.

Photo by author

As you can see, the UNNEST operator transforms the array into a denormalized format, ready to be consumed by MS Excel or Google Sheets. From 1 row nesting 2 cuisine labels for Ho Jiak restaurant at Haymarket, we now end up with 2 separate rows.

Aggregate elements into arrays with ARRAY_AGG

What if we have a denormalized table beforehand and need to aggregate all cuisine labels into array format so that you end up with fewer rows?

Let’s save the result of the above query into a BigQuery table named “restaurant_cuisine_unnested”. Next, we will use the ARRAY_AGG operator with GROUP BY to bring it back to the array format.

Photo by author

Voilà! Here you go, things are nicely re-packaged to normal with 4 rows for 4 restaurants. Do you notice something different regarding the order of cuisine for each row? (Hint: Notice how Casual now appears above Japanese for Chaco Bar)

Yes, since we have opted to use “ORDER BY cuisine” with ARRAY_AGG, all cuisine labels are now sorted in alphabetical order. This order will be preserved if we save it as a permanent table.

Count the elements in an array with ARRAY_LENGTH

It’s time to turn things up a notch and see what else we can do with arrays. Let’s count how many cuisine labels each restaurant is associated with.

Photo by author

Query Array with UNNEST and WHERE IN

What if we want to list all related information about casual restaurants?

To solve this question, we will have to go through each element of the arrays one by one and filter only cuisine containing “Casual”. But an array column behaves differently from a normal column. You can’t go through and filter elements inside an array without first unnesting it. Keeping that in mind, I have included 2 slightly different queries that give you the same result in 2 different formats, depending on what you want.

Option 1. If you want to obtain a denormalized query result without any array (to consume in Excel or Google Sheets)

Photo by author

Option 2. If you want your cuisine column nicely packed into an array (to save storage costs and query time)

Note: The below query has been kindly provided by Michael Entin. I decide to replace it with the original query since it’s way simpler and more concise. Once again, thanks @mentin for enlightening me!