BigQuery for Data Science
By: David Ten
Tags: Machine Learning, Google Cloud Platform, Data Science, Pandas, Bigquery

BigQuery for Data Science

What is BigQuery?

One of the perks of using Google Cloud Platform (GCP) is having BigQuery, Google’s cloud hosted data warehouse solution at your disposal. BigQuery gives GCP users access to the key features of Dremel, Google’s very own internal data warehouse solution. Under the hood Dremel stores data in columnar format and uses a tree architecture to parallelise queries across thousands of machines, with each query scanning the entire table.

So, what is so great about that? With BigQuery you can run SQL queries on a table with billions of rows and get the results in seconds! Although thousands of Google machines process the data, each query only takes up a small amount of compute time, so it only costs $5.00 per TB of data processed. Best of all, Google takes care of infrastructure management.

Queries which used to take hours or longer on traditional databases can now be processed in a matter of seconds. This enables analysts to answer business questions rapidly, build dashboards on aggregated data, etc.

What about BigQuery and Data Science?

So if your data is now in a BigQuery data warehouse, does it speed up data science? The answer is it depends. While BigQuery is good at querying large amounts of data, data processing and transformation is limited to basic SQL-like functions. Complex data processing will have to be done outside of BigQuery, which brings us to the next problem; getting data out of BigQuery is slow. Specifically, moving the data into a pandas or R dataframe is slow.

You start to notice this when working with hundreds of thousands of rows or more. In python, the Google driven library google-cloud-bigquery and the community driven pandas-gbq library allow users to transfer the BigQuery result to pandas. Both face data transfer problems, as highlighted in this GitHub issue.

For example, running a query data from the public New York Yellow Taxi 2018 dataset returning 2 million lines out of 63 million only takes 18 seconds on BigQuery:

However loading the result into a pandas dataframe via the to_dataframe() function takes almost 6 minutes:

It can be frustrating to wait for minutes to load the result into pandas when the query only took seconds. In some cases, the data scientist has access to both the traditional SQL database and the BigQuery tables and getting data from the traditional SQL database into pandas can actually be faster as there wasn’t that much data to scan in the first place.

In R, the bigrquery package faces the same data transfer issue as well, although it is faster as the system for downloading data has been written to improve performance.

Workarounds

There are a few ways to work around this, depending on whether you are doing exploration or building production machine learning pipelines.

Exploratory

Typically, when starting to work on a new dataset, data scientists use Jupyter notebooks to rapidly explore the data and build preliminary models. Multiple BigQuery calls might be made and refined as the data is explored.

1. Limit Query Size

The simplest solution is to limit the query result either by limiting the number of rows returned or filtering and aggregating the results in order to reduce the amount of data being transferred. BigQuery isn’t really meant to transfer huge amounts of data anyway. However, this depends on the number of columns that you have and in the era of big data, you might not get a representative sample of your dataset.

2. Dump to csv

Writing the query result to csv first and then reading csv result into pandas can actually speed things up, by up to 10 times! To be exact, due to quirks in the API you have to first export the query result to a table and then export the table to csv in a Cloud Storage Bucket. This way, it is possible to only query the data once and then read from your storage, giving you repeatable results. This method is a bit clunky and there is some overhead to manage the different csv outputs, buckets and tables.

Using this method, writing to csv in a Storage Bucket and reading from csv only takes about half a minute each:

Export to GCS:

Read from GCS:

3. BigQuery Storage

As I was writing this, Google has released the beta version of BigQuery Storage, allowing fast access to BigQuery data, and hence faster download into pandas. This seems to be an ideal solution if you want to import the WHOLE table into pandas or run simple filters. However, the tradeoff is that SQL type queries no longer work. Columns can be selected while rows can be filtered, but only by comparing a column to a constant value. If you need to sample the data then you have to rely on filtering by a value in a column.

In the example below, we try out the BigQuery Storage API. Since SQL-like statements don’t work, we filter the pickup_location_id to ‘48’ to get about 2 million rows in return. Getting the data into pandas took about 100 seconds, faster than loading directly from BigQuery but slower than dumping into csv.

However with BigQuery Storage there is the possibility of reading in the data with multiple streams, increasing the transfer speed.

4. BigQuery ML

BigQuery ML is an effort by Google to allow BigQuery users to build machine learning models using SQL queries. Instead of bringing data out of BigQuery and then building a model, the model is brought to the data instead. While the goal is for everything to be done in BigQuery, I find the solution very limited as complex data transformations cannot be performed with SQL and only linear and logistic regression models are available.

Production

In production a significant chunk of the data will have to be processed and robust solutions are needed. While BigQuery Storage could be a solution to run on pandas in production, it would probably be better to avoid pandas altogether due to performance issues and build more robust processing pipelines instead.

1. TensorFlow

BigQuery stores structured data. Why TensorFlow? Actually, the TensorFlow framework can handle structured data as well with models such as linear regression, logistic regression, boosted trees, etc. The BigQueryReader class allows Tensorflow to access BigQuery.

2. Spark

Like Tensorflow, BigQuery also has connectors to Spark, allowing the use of libraries like H2O.ai’s Sparkling Water. Unfortunately, if you’re using pyspark, the data will have to be dumped to Cloud Storage first before Spark can read it.

Conclusion

While BigQuery enables rapid queries with SQL syntax, extracting the data for data science can be tedious. If limiting the query size does not work for me, I personally prefer to dump to csv and read from there as it fast and allows repeatable analysis.

Further improvements to the google-cloud-bigquery library would be an interface to Spark/Dask for lazy reading of data as well as the ability to write dplyr like transformations of the data instead of writing SQL queries to access data in BigQuery.