PIVOT in BigQuery
By: Lak Lakshmanan
Tags: Sql, Bigquery, Google Cloud Platform

PIVOT in BigQuery

Sometimes, you might want to reformat a table result so you have separate columns for each unique value. This is called a Pivot table — normally, it’s only a display function supported by BI tools. However, it can occasionally be helpful to create the pivot table in SQL. Here’s how to use the PIVOT operator in Google BigQuery to create a pivot table.

Image by Gerd Altmann from Pixabay

What does Pivot do?

Pivot changes rows to columns. For example, suppose we have a table of flights and flight delays like this:

And we want output where we obtain the average departure delay for each airport-airline pair, where the airline codes are columns:

We’d do a PIVOT. Note that if we simply want the average departure delay and are okay with the averages being on different rows, we could simply do:

SELECT

airline,

departure_airport,

AVG(departure_delay)

FROM `bigquery-samples.airline_ontime_data.flights`

GROUP BY 1, 2

and obtain:

It’s the fact that we need it laid out as a 2D table that requires a PIVOT.

Syntax of PIVOT

The Pivot operator in BigQuery needs you to specify three things:

from_item that functions as the input. The three columns (airline, departure_airport, departure_delay) from the flights table is our from_item.

aggregate since each cell of the output table consists of multiple values. Here, that’s the AVG of the departure_delay

pivot_column, the column whose values form the columns in the output table. Unfortunately, this needs to be a constant — you can’t put a query in there.

Let’s look at a couple of examples.

Flights table PIVOT

We want departure delays by airport and airline. The pivot query is:

SELECT * FROM

(

-- #1 from_item

SELECT

airline,

departure_airport,

departure_delay

FROM `bigquery-samples.airline_ontime_data.flights`

)

PIVOT

(

-- #2 aggregate

AVG(departure_delay) AS avgdelay

-- #3 pivot_column

FOR airline in ('AA', 'KH', 'DL', '9E')

)

Note the three parts to pivot the departure delays into a table organized by departure-airport and airline code. Because the pivot_column needs to be a constant, we have to list them explicitly.

Result:

The reason for the prefix avgdelay is that we called the aggregate avgdelay.

Stack Overflow PIVOT

Let’s create a pivot of the number of questions with a given tag by year

SELECT

tag,

EXTRACT(YEAR from creation_date) AS year,

COUNT(*) AS n

FROM

`bigquery-public-data.stackoverflow.posts_questions`,

UNNEST(SPLIT(tags, '|')) AS tag

WHERE tags IS NOT null

GROUP BY 1, 2

where the rows are the tags and the columns are the years:

SELECT * FROM

(

-- #1 from_item

SELECT

tag,

EXTRACT(YEAR from creation_date) AS year

FROM

`bigquery-public-data.stackoverflow.posts_questions`,

UNNEST(SPLIT(tags, '|')) AS tag

WHERE tags IS NOT null

)

PIVOT

(

-- #2 aggregate

COUNT(*) AS n

-- #3 pivot_column

FOR year in (2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020)

)

Dynamically generating pivot column values

What if you can’t list all the possible values or the pivot column beforehand? One solution is to use scripting. Here, I create a variable called airlines with that prints out the string in the right format (try it out):

DECLARE airlines STRING;

SET airlines = (

SELECT

CONCAT('("', STRING_AGG(DISTINCT airline, '", "'), '")'),

FROM `bigquery-samples.airline_ontime_data.flights`

);

Then, I can use EXECUTE IMMEDIATE to run the actual query:

EXECUTE IMMEDIATE format("""

SELECT * FROM

(

SELECT

airline,

departure_airport,

departure_delay

FROM `bigquery-samples.airline_ontime_data.flights`

)

PIVOT

(

AVG(departure_delay) AS avgdelay

FOR airline in %s

)

ORDER BY departure_airport ASC

""", airlines);

This results in:

Enjoy!