BigQuery SQL performance tips | Towards Data Science

Google BigQuery, like other modern hyper-scale data platforms, has a different architecture to what many data professionals and data scientists are used to; it stores its data in columns instead of rows (referred to as a column-store), and processes SQL queries in a fully distributed architecture.
It is these unique properties that enable the platform to achieve such high levels of query performance.
If, like me, you came to BigQuery from an MS SQL background, a lot of what I knew in regards to writing efficient SQL queries, actually no longer applied. There are subtle differences that I have learned whilst working with BigQuery nearly every day, on large-scale client projects over recent years.
In this article, I share my top 5 key learnings.
I hope this helps you to write faster queries, and lower your costs :)
To help illustrate these tips, I will be using the publicly available cycle ride data from the London public cycle hire scheme. You can access this by following these instructions.
cloud.google.com
This comprises a stations table, containing 718 stations.
And a rides table, containing ~24M individual cycle rides.
I imagine that you are familiar with writing SQL statements like this one, to limit the number of rows returned from a (typically large) table:
select top 100 *from blog_perf.rides
Even if the table is very large, this query will execute very quickly; the database engine simply retrieves only the first 100 rows it finds and does not need to, therefore, perform a full scan on the table.
Often I see people do this in BigQuery (at first, myself included!):
select *from blog_perf.rides limit 100
This also returns 100 rows. So what’s the big deal?
Well, BigQuery handles these limit clauses differently; in the above example, despite my limit 100, BigQuery does in fact perform a full table scan of my rides table.
What’s happening?
BigQuery is applying the limit clause after the main query has been executed; only then, is the limit applied to reduce the rows returned.
So be careful with limit, and certainly don’t use it as a way to reduce scan times, or scan costs.
Instead, restrict your data in your joins and where clause.
Tip: Resist the urge to use limit 10 to inspect the top 10 rows of a table during data discovery; if that table is large, you will be charged for a full table scan! Instead, use the handy preview feature in BigQuery. Just click on the table name, and then click on the Preview tab to see the top 100 rows. For free!
Bonus tip: When in the SQL window in the BigQuery UI, hold the control key, and click on a table name; the above Schema, Details and Preview tabs are displayed for that table.
One factor that impacts performance in BigQuery is the volume of data scanned when your SQL queries are executed. Note too, data scan is also tied to how much you are charged (together with how much data your store).
BigQuery offers a super-helpful feature in the UI that actually tells you, before a SQL query is executed, how much data that query will scan. Clever, hey?
I use this feature a lot, and I would encourage others to use this too. It’s a great way to learn about what is happening “under the hood”, especially when you start to work with partitioned tables.
To show you how useful it is, let’s see what it estimates for a full scan of our rides table. I just type my SQL in the BigQuery UI, and in real-time it reveals:
This query will process 2.8GB when run
As highlighted below. The green tick indicates my SQL is valid.
Now, I add my limit 100.
And here we see — no change in the amount of data scanned!
Another really useful feature, is you can access these data scan metrics for other queries executed in BigQuery. To do this:
Click Query History in the left sidebarBigQuery displays my queries executed, the most recent at the topClick on the query you want to explore to see the results.
Click Query History in the left sidebar
BigQuery displays my queries executed, the most recent at the top
Click on the query you want to explore to see the results.
Here is mine for the query above:
Note this tells me:
Duration 0.5 secs Bytes processed 2.77 GB
Now, applying the limit 100, whilst it does not affect the data scanned, it does reduce the amount of data returned to the client. This has a dramatic impact on the duration for the query.
If I remove the limit 100, the duration increases from 0.5 secs to 28.3 secs, more than a factor of 50!
As well as your own queries, you can also see these metrics for other queries executed on this BigQuery instance; really helpful for example, if you are troubleshooting a slow running or expensive process.
To do this, just click on the Project history tab:
In my previous post, below, I warned against the use of select * in your SQL code.
towardsdatascience.com
Well, performance on BigQuery is yet, another reason, not to use select *.
Why?
Well, limiting your SQL to only the columns that you need reduces the amount of data BigQuery scans (reducing cost), and if your query is returning data to a client, or say loading a table, it will also reduce the execution time.
So, my previous query below scanned 2.77 GB of data and ran for 28.3 secs.
select *from blog_perf.rides
If I limit this to just a single column:
select r.start_date from blog_perf.rides as r
This reduces the data scanned to 185.92 MB and executed in just 7.5 secs, reducing the data scanned by a factor of nearly 15, and the execution time by a factor of 4.
Consider a query where we select all station names that have had at least one journey.
One way to achieve this is to use the exists predicate:
select s.name from blog_perf.stations as s where exists( select * from blog_perf.rides as r where s.id = r.start_station_id )
Note the select *
Should we, therefore, change this select * to a select 1 like this:
select s.name from blog_perf.stations as s where exists( select 1 from blog_perf.rides as r where s.id = r.start_station_id )
What do you think?
Well, it turns out that this is one occasion where actually, a select * is OK. In both queries, the data scanned and execution times are identical.
In BigQuery’s distributed architecture, any order by clauses that occur in the middle of a SQL statement, greatly impact performance. This is because, in order to sort the rows, BigQuery must marshall data from distributed nodes in order to sort the data, before continuing to broadcast data back out.
Instead, if you do need to sort data, keep your order by statements on the outermost level of your SQL query.
Here’s an example.
For each starting station, return the bike id that made the most trips, and order by the total number of trips made by each bike.
This scanned 1 GB of data and took 14.3 secs to execute.
with bike_ridesas(   select r.start_station_id,          r.start_station_name,          r.bike_id,          count(*) as ride_count   from blog_perf.rides r   group by 1,2,3),top_bikesas(   select br.start_station_id,          br.start_station_name,          br.bike_id   from bike_rides br   where br.ride_count =    (      select max(ride_count)      from bike_rides brm      where brm.start_station_id = br.start_station_id   ))select tb.start_station_name,       tb.bike_id,       count(*) as bike_ride_countfrom top_bikes tb  inner join blog_perf.rides r  on r.bike_id = tb.bike_idgroup by 1,2order by 3 desc
Now, let’s add an order by in some of the subqueries, like this:
with bike_ridesas(   select r.start_station_id,          r.start_station_name,          r.bike_id,          count(*) as ride_count   from blog_perf.rides r   group by 1,2,3   order by 1,2,3),top_bikesas(   select br.start_station_id,          br.start_station_name,          br.bike_id   from bike_rides br   where br.ride_count =    (      select max(ride_count)      from bike_rides brm      where brm.start_station_id = br.start_station_id   )   order by 1,2,3)select tb.start_station_name,       tb.bike_id,       count(*) as bike_ride_countfrom top_bikes tb  inner join blog_perf.rides r  on r.bike_id = tb.bike_idgroup by 1,2order by 3 desc
If we click on Execution details, this is what the first query looked like compared with the second (the second has the additional order by clauses):
Note the slot time increased by over 1 minute, and the bytes shuffled by 0.5 GB.
When constructing SQL statements, always start with large tables first, filtering these as much possible in your where clause, before joining to (ideally, smaller) tables.
This greatly reduces the amount of shuffling BigQuery needs to perform in its distributed architecture (the amount of data passed between distributed nodes).
The query optimiser will try to do this for you, but to be on the safe side, I would recommend keeping this in mind.
Here’s an example:
We want to count the number of rides that started at a station near a park, together with the date that the station was installed.
The efficient way to do this is:
select s.name,       s.install_date,       count(*) as ride_countfrom blog_perf.rides r inner join blog_perf.stations s  on s.id = r.start_station_id where s.name like ‘%Park%’group by 1,2
Note we start with the largest rides table, and then join to the smaller station table, which is further reduced in size by filtering on the name to limit those near a park.
This scans 184.2 MB of data and took 0.4 secs to execute.
An inefficient way to do this, would be like this:
select s.name,       s.install_date,       count(*) as ride_countfrom blog_perf.rides r inner join blog_perf.stations s  on s.id = r.start_station_id where r.start_station_name like ‘%Park%’group by 1,2
The only change here is to filter the larger ride table. But the net result, is a larger set of stations is shuffled to nodes containing rides in order to produce the final counts.
This scans 860 MB — nearly a 5 fold increase, just from this, seemingly trivial, change.
Working with modern, column-store, distributed data platforms like BigQuery, demands some new knowledge in how to ensure SQL is written in a performant, and cost-efficient manner. Many tricks and tips learned in the past, don’t necessarily apply; in fact, as we have seen, they can actually work against you.
I hope you have found this article helpful. In the next article, I’ll share some tips for working with shared and partitioned tables.
1. Learn more about Ancoris Data, Analytics & AI