How to Work with Nested Data in BigQuery | by Christianlauer | Towards Data Science

Cloud data lakes and warehouses are on the rise — one example is Google’s BigQuery. BigQuery is the most powerful with denormalized data. Instead of working with traditional schemas like star or snowflake schemas, you should denormalize your data and use nested and recurring columns. These columns can preserve relationships without degrading performance as relational or normalized schemas do [1].
BigQuery supports loading and querying nested and recurring data from source formats that support object-based schemas for example JSON.
The address column contains an array of values. The different addresses in the array are the recurring data. The different fields within each address are the nested data.
For an example walkthrough with BigQuery I used the open dataset planet_features in the geo_openstreetmap dataset. Here, the data is stored in a nested format, so let’s take a look:
SELECT * FROM `bigquery-public-data.geo_openstreetmap.planet_features` LIMIT 1000
This is the result in the following output:
The result looks good at first sight. Unlike classical relational databases, I can use arrays and save additional columns. Furthermore, the whole thing works with a super performance. Now, I would like to find a certain supermarket in Germany — should be done with an easy where clause — right? Almost. Here, you need the magic of Unnest,
SELECT * FROM   `bigquery-PUBLIC-   data.geo_openstreetmap.planet_features` WHERE  'Netto' IN        (               SELECT value               FROM   unnest(all_tags)) AND    (               'addr:country', 'DE') IN        (               SELECT (KEY, value)               FROM   unnest(all_tags)) AND    (               'addr:city', 'Hamburg') IN        (               SELECT (KEY, value)               FROM   unnest (all_tags));
which results in the desired output:
With the function of unnest you will also be able to flatten the data and so the output of the query:
SELECT osm_id,tagsFROM bigquery-public-data.geo_openstreetmap.planet_features,UNNEST(all_tags) as tags limit 100
This is probably almost the most important you have to know when working with nested data in BigQuery and want to query some data. But what if you want to use the data for further ETL processes, store the data in relational databases or need the key-value-pairs as attributes for your classifier? Here, you want the data not to flatten — because this will result in duplicate rows — here you want the key-value-pairs in the array as new columns:
SELECT        (               SELECT osm_id) osmid,        (               SELECT value               FROM   Unnest(all_tags)               WHERE  KEY = "Address") AS address,        (               SELECT value               FROM   Unnest(all_tags)               WHERE  KEY = "name") AS NAME,        (               SELECT value               FROM   Unnest(all_tags)               WHERE  KEY = "opening_hours") AS opening_hours,        (               SELECT value               FROM   Unnest(all_tags)               WHERE  KEY = "organic") AS organic,        (               SELECT geometry) AS geometry, FROM   bigquery-PUBLIC-data.geo_openstreetmap.planet_features WHERE  (               'Edeka' IN               (                      SELECT value                      FROM   unnest(all_tags))        OR     'Rewe' IN               (                      SELECT value                      FROM   unnest(all_tags))        OR     'Netto' IN               (                      SELECT value                      FROM   unnest(all_tags))) AND    (               'addr:country', 'DE') IN        (               SELECT (KEY, value)               FROM   unnest(all_tags)) - AND   (               'addr:city', 'Hamburg') IN        (               SELECT (KEY, value)               FROM   unnest(all_tags));
After I came into contact with nested data for the first time — I have to admit that I didn’t know what to do with it at first and probably had the same facial expression as Mark Wahlberg in the movie The Happening.
But I realized in the new world, such data formats are normal and the way to the world of unstructured data. New systems offer extremely high computing power and fast results through column-based databases. With the above examples, you should be able to query and process most of your nested data use cases very well.
[1] Google, Specifying nested and repeated columns