The Best Way to Generate Indices in BigQuery | by Jye Sawtell-Rickson | Towards Data Science

When you stumble upon your first task of creating a histogram within BigQuery you might approach it with a simple GROUP BY, something like:
SELECT    hour,    COUNT(*) AS num_accessesFROM    accessesGROUP BY    hourORDER BY    hour
This is fine, it does a decent job and would earn you a pat on the back from your friendly colleague. What it doesn’t do though, is handle the case of missing hours. For example, you might have accesses throughout most of the day but at 3 am no one accesses your site, sorry. This would leave a gap in your histogram which just isn’t very nice, is it? And in some cases, can look downright silly.
Instead, we can use a variety of techniques to get that extra hour, my favourite of which combines the harmonious duo of UNNEST and GENERATE_ARRAY.
The general idea is to generate an array of all the possible values, convert that array into rows and then use those rows as an index to join your results onto — elementary, you say! Let’s try this:
SELECT * FROM UNNEST(GENERATE_ARRAY(0, 23)) AS hour
This query will generate an array with the numbers 0, 1, ..., 23 to pass to UNNEST which takes those values and puts them in rows, finally labeled as hour.
With that index we can then do a left join (we want to keep the null rows in the index else all would have been for naught). Our final query looks something like this:
WITH index AS (SELECT * FROM UNNEST(GENERATE_ARRAY(0, 23)) AS hour),hourly_counts AS (    SELECT        hour,        COUNT(*) AS num_accesses    FROM        accesses    GROUP BY        hour    )SELECT   hour,   COALESCE(num_accesses, 0) AS num_accessesFROM    index    LEFT JOIN hourly_counts USING (hour)ORDER BY    hour
Instead of the convenient predefined bounds we used above, you might have variable bounds. An example would be if you’re trying to produce a histogram of the number of accesses in a day. Again, you might try to apply a GROUP BY, but you won’t have the missing indices.
WITH daily_accesses AS (    SELECT        day,        COUNT(*) AS num_accesses    FROM        accesses    GROUP BY        day    )SELECT    num_accesses,    COUNT(*) AS num_daysFROM    daily_accessesGROUP BY    num_accessesORDER BY    num_accesses
No good! So let’s try generating the index again, but this time using variable bounds, since the num_accesses on any given day is unconstrained.
SELECT    *FROM    UNNEST(GENERATE_ARRAY(        (SELECT MIN(num_accesses) FROM daily_accesses)mi,        (SELECT MAX(num_accesses) FROM daily_accesses)ma        )) AS num_accesses
From our daily accesses table we’ve pulled out the minimum and maximum values, which gives us the lower and upper bound, respectively. Combining these all together we get:
WITH daily_accesses AS (    SELECT        day,        COUNT(*) AS num_accesses    FROM        accesses    GROUP BY        day    ),day_counts AS (    SELECT        num_accesses,        COUNT(*) AS num_days    FROM        daily_accesses    GROUP BY        num_accesses    ),index AS (    SELECT        *    FROM        UNNEST(GENERATE_ARRAY(            (SELECT MIN(num_accesses) FROM daily_accesses)mi,            (SELECT MAX(num_accesses) FROM daily_accesses)ma            )) AS num_accesses    )SELECT    num_accesses,    COALESCE(num_days, 0) AS num_daysFROM    index    LEFT JOIN day_counts USING (num_accesses)ORDER BY    num_accesses
A little more involved than the previous example, but the principle remains the same.
Finally, you’re likely to come across a date index during your data journey. You could generate a number using GENERATE_ARRAY and then use something like DATE_ADD to get the dates, but Bigquery has a built-in function GENERATE_DATE_ARRAY.
Consider now if we want to get the daily number of accesses to our website for 2020 (it’s review time!). Since it has been a year with a lot of variability, there may be days without any accesses.
Again, our basic query:
SELECT    date,    COUNT(*) AS num_accessesFROM    accessesGROUP BY    dateORDER BY    date
And we create our index like:
SELECT * FROM UNNEST(GENERATE_DATE_ARRAY(    '2020-01-01',    CURRENT_DATE()    )) AS date
Which will create one date for each day up until today in 2020. Giving the final query.
WITH index AS (    SELECT * FROM UNNEST(GENERATE_DATE_ARRAY(        '2020-01-01',        CURRENT_DATE()        )) AS date   ),daily_counts AS (    SELECT        date,        COUNT(*) AS num_accesses    FROM        accesses    GROUP BY        date    )SELECT   date,   COALESCE(num_accesses, 0) AS num_accessesFROM    index    LEFT JOIN daily_counts USING (date)ORDER BY    date
Throughout the examples so far, we’ve assumed the interval is 1 (or 1 DAY), but it’s perfectly reasonable to use different step sizes such as every seven days. To achieve such things you can pass an additional parameter to the GENERATE function which will then return an adjusted index. In this case, you’ll need to be a little more tricky in how you join your underlying data, but it’s achievable for the determined data wrangler!
Hopefully this simple approach to generating indices can help you out next time you need a well defined index. As always, check out the Bigquery docs if you’d like to learn more.
Let me know if you use any other interesting or convenient methods!