UNPIVOT multiple columns into tidy pairs with BigQuery and a SQL UDF | by Felipe Hoffa | Towards Data Science

Important update: I left Google and joined Snowflake in 2020 — so I’m unable to keep my older posts updated. If you want to try Snowflake, join us — I’m having a lot of fun ❄️.
As an example of non-tidy data, we can see how the Novel Coronavirus (COVID-19) Case (provided by JHU CSSE), and the Apple Mobility Trends Reports tables look:
We don’t want multiple columns, each for one date. We want to have (date, value) pairs. Since this problem seems so common, I wrote two BigQuery persistent UDFs to solve this:
fhoffa.x.unpivot()
fhoffa.x.cast_kv_array_to_date_float()
Let’s review how they work.
Just give unpivot() a full row, and the regex of how the name of each of the columns to unpivot look.
With the Apple tables:
SELECT a.geo_type, region, transportation_type, unpivottedFROM `fh-bigquery.public_dump.applemobilitytrends_20200414` a  , UNNEST(fhoffa.x.unpivot(a, '_2020')) unpivotted
With the JHU tables:
SELECT province_state, country_region, unpivottedFROM `bigquery-public-data.covid19_jhu_csse.confirmed_cases`  a  , UNNEST(fhoffa.x.unpivot(a, '_[0-9]')) unpivotted
That’s so much better, but we are not done. How do we transform these values into dates and numbers?
We can re-cast our unpivotted columns with cast_kv_array_to_date_float().
What’s even more “infuriating” when casting these columns to dates is that they use different formats for encoding dates. You don’t have to worry as the UDF can take the date format as an input too.
For example, with the Apple tables:
SELECT a.geo_type, region, transportation_type, unpivotted.*FROM `fh-bigquery.public_dump.applemobilitytrends_20200414` a  , UNNEST(fhoffa.x.cast_kv_array_to_date_float(fhoffa.x.unpivot(a, '_2020'), '_%Y_%m_%d')) unpivotted
And with the JHU tables:
SELECT province_state, country_region, unpivotted.*FROM `bigquery-public-data.covid19_jhu_csse.confirmed_cases`  a  , UNNEST(fhoffa.x.cast_kv_array_to_date_float(fhoffa.x.unpivot(a, '_[0-9]'), '_%m_%d_%y')) unpivotted
See? These results look way tidier than the starting tables.
Once we have these 2 UDFs, applying them to other tables becomes really easy:
SELECT county_fips_code, county_name, state, state_fips_code, unpivotted.*FROM `bigquery-public-data.covid19_usafacts.confirmed_cases` a  , UNNEST(fhoffa.x.cast_kv_array_to_date_float(fhoffa.x.unpivot(a, '_[0-9]'), '_%m_%d_%y')) unpivotted
Check my previous post about Persistent UDFs in BigQuery:
medium.com
The source code for these 2 UDFs is:
CREATE OR REPLACE FUNCTION fhoffa.x.unpivot(x ANY TYPE, col_regex STRING) AS ((  # https://medium.com/@hoffa/how-to-unpivot-multiple-columns-into-tidy-pairs-with-sql-and-bigquery-d9d0e74ce675  SELECT    ARRAY_AGG(STRUCT(     REGEXP_EXTRACT(y, '[^"]*') AS key   , REGEXP_EXTRACT(y, r':([^"]*)\"?[,}\]]') AS value   ))  FROM UNNEST((    SELECT REGEXP_EXTRACT_ALL(json,col_regex||r'[^:]+:\"?[^"]+\"?') arr    FROM (SELECT TO_JSON_STRING(x) json))) y));CREATE OR REPLACE FUNCTION fhoffa.x.cast_kv_array_to_date_float(arr ANY TYPE, date_format STRING) AS ((  # https://medium.com/@hoffa/how-to-unpivot-multiple-columns-into-tidy-pairs-with-sql-and-bigquery-d9d0e74ce675  SELECT ARRAY_AGG(STRUCT(SAFE.PARSE_DATE(date_format, key) AS date, SAFE_CAST(value AS FLOAT64) AS value))  FROM UNNEST(arr)));
The secret motor behind this function: Transforming a whole row into JSON with TO_JSON_STRING() and then doing a REGEXP_EXTRACT_ALL over it.
My previous solution to UNPIVOT in BigQuery has received than 5k views on Stack Overflow:
stackoverflow.com
Once I write documentation for these functions, and we settle on their definitive name — I’ll submit them to our shared repository with community UDFs (bqutil).
github.com
Check Google’s public dataset program, featuring an increasing collection of COVID-19 related datasets in BigQuery:
cloud.google.com
I’m Felipe Hoffa, a Developer Advocate for Google Cloud. Follow me on @felipehoffa, find my previous posts on medium.com/@hoffa, and all about BigQuery on reddit.com/r/bigquery.