Introduction to Arrays in BigQuery | by ChiefHustler | Towards Data Science

I work with BigQuery on a daily basis, and I have to say its speed and ease of use are mind-blowing.
Why do I like BigQuery? Three reasons:
It’s serverless and fully managed. I don’t have to think about anything but my queries.It’s super-fast, thanks to a highly efficient compressed columnar storage.It has a modern user interface: I can do everything from the browser, no clunky install.
It’s serverless and fully managed. I don’t have to think about anything but my queries.
It’s super-fast, thanks to a highly efficient compressed columnar storage.
It has a modern user interface: I can do everything from the browser, no clunky install.
Just to give you an example of the performance, here’s a table with 18.5M rows:
Let’s run a simple regex query on it to extract the title and body of all the questions where the title mentions one of the following programming languages: python, rstats, sql:
select title, body from `bigquery-public-data.stackoverflow.posts_questions` where regexp_contains(title, r"\b(python|rstats|sql)\b")
Here’s the output:
I don’t know about you, but 21 seconds to process 26.9 GB...I’m impressed. Want to know how much that cost? BigQuery charges 5$ per TB processed, so that query cost 26.9 / 1000 * 5 = 0.1345$.
With that, let’s dive straight into working with arrays in BigQuery.
You probably guessed it by now, we’re gonna be counting words occurring in StackOverflow questions. But before we get to the practical example, I’d like to cover the basics of working with arrays in BigQuery.
In the prototyping phase, it can be useful to create small arrays to play with. For example:
select 'Engineering' as role, 'Mike' as name, ['Java', 'C#'] as languages union all select 'Data science' as role, 'Vlad' as name, ['python', 'SQL', 'R'] as languages union all select 'Front-end' as role, 'Guillaume' as name, ['javascript', 'C#', 'HTML', 'CSS'] as languages
This gives us the following output:
Notice how the arrays are nested vertically inside each row.
If we want to turn this nested structure into a regular flat table, we need to unnest the arrays:
with T0 as (select 'Engineering' as role, 'Mike' as name, ['Java', 'C#'] as languages union all select 'Data science' as role, 'Vlad' as name, ['python', 'SQL', 'R'] as languages union all select 'Front-end' as role, 'Guillaume' as name, ['javascript', 'C#', 'HTML', 'CSS'] as languages) SELECT role, name, languages from T0 CROSS JOIN UNNEST(T0.languages) as languages
The CROSS JOIN UNNEST code performs what's known as a correlated join, meaning it keeps track of which row each array was associated with, so as not to perform a full cross join needlessly.
We now get the more familiar flat and repeated structure:
Let’s say we want to know what is the last language for each programmer. There are two ways to index in an array in BigQuery: 0-based indexing (python style) with OFFSET or 1-based indexing (R style) with ORDINAL.
We’ll use ORDINAL as an example, along with ARRAY_LENGTH to retrieve the length of each array. Using the same definition of T0 as above:
SELECT role, name, languages[ORDINAL(ARRAY_LENGTH(languages))] as last_language from T0
With the basics out of the way, let’s jump into a concrete use case for arrays.
We’d like to have a way to count the frequency of every word in the questions titles, and if possible, a way to capture correlations between words and question tags.
A key function to turn strings into arrays of words is the split function. Here's an example:
select title, tags, split(title, ' ') as words from `bigquery-public-data.stackoverflow.posts_questions` limit 10
Interestingly, we see in the image above that we have a string of tags. Let’s turn that into an array as well:
select title, split(tags, '|') as tags, split(title, ' ') as words from `bigquery-public-data.stackoverflow.posts_questions` limit 10
So each row is going to contain 2 arrays:
And now we’re going to cross-join/unnest the two arrays at once, and filter out uninteresting words, i.e. words less than 2 characters long and some common stop words:
with t0 as (select id, title, split(tags, '|') as tags, split(title, ' ') as words from `bigquery-public-data.stackoverflow.posts_questions`) select id, tag, word from t0 cross join unnest(t0.words) as word cross join unnest(t0.tags) as tag where length(word) > 1 and lower(word) not in ('me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', "isn't", 'are', 'was','were', "aren't", "wasn't", "weren't", 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 'can', "can't", 'will', "won't", 'just', "don't", 'should', "shouldn't", 'now', 'get')
This query took 24 seconds to run, and just like that, we have obtained the cartesian product of all related tags and words:
Now we can easily ask what words are most represented for a given tag, e.g. python (assume the output of the previous step has been saved in table so, inside dataset temp):
select word, count(*) as frequency from `temp.so` where tag='python' group by word order by frequency desc
And here’s what we get:
Or we can go the other way around and ask what tags are associated with a particular word, e.g. flask:
select tag, count(*) as frequency from temp.so where word = 'flask' group by tag order by frequency desc
And we get:
A tf-idf analysis would be useful here to single out what is really specific to python as opposed to other tags. Although outside the scope of this overview, it would make for a great side-project.
Arrays constitute a great feature of BigQuery, and there are other goodies that I looking forward to writing about. For my money, BigQuery is one of the most impressive big data analytics tool out there!