Optimizations in BigQuery
Tags: Data Warehouse, Cloud, Datacouch, Google Cloud Platform, Bigquery

Introduction

Did you know Google is named as a Leader in Cloud Database Management Systems (BigQuery) by Gartner in the 2020 Magic Quadrant Q4. BigQuery is a serverless, highly scalable, and cost-effective data warehouse which is provided by Google. Using this one can analyze TBs of data in seconds that too by just typing SQL-like queries. Surely BigQuery processes the TBs of data in seconds but some optimization can make it to Milliseconds. So let’s understand how we can optimize our system and convert these seconds of analytics into milliseconds.

Learning Objectives

What is BigQuery? BigQuery Architecture Advised Optimization in BigQuery Hands-On Exercise of Some Optimization Concepts

What is BigQuery?

It is a Serverless data warehouse service

It is also highly scalable service provided by the google

It’s cost-effective multi-cloud data warehouse designed for business agility

Democratize insights with a secure and scalable platform with built-in machine learning

Power business decisions from data across clouds with a flexible, multi cloud analytics solution

Analyze petabytes of data using ANSI SQL at blazing-fast speeds, with zero operational overhead

Note: If you are not a reading person then also we have some great content for basic introductory knowledge about BIgQuery. Visit our YouTube channel and watch our wonderful videos about BigQuery

BigQuery Architecture

(Source: Google Cloud)

BigQuery’s serverless architecture decouples storage and compute and allows them to scale independently on demand

This structure offers both immense flexibility and cost controls for customers because they don’t need to keep their expensive compute resources up and running all the time

Under the hood, BigQuery employs a vast set of multi-tenant services driven by low-level Google infrastructure technologies like Dremel, Colossus, Jupiter and Borg

(Source: Google Cloud)

Dremel turns SQL queries into execution trees. The leaves of the tree are called slots and do the heavy lifting of reading data from storage and any necessary computation. The branches of the tree are ‘mixers’, which perform the aggregation

Dremel dynamically apportions slots to queries on an as-needed basis, maintaining fairness for concurrent queries from multiple users. A single user can get thousands of slots to run their queries

Colossus also handles replication, recovery (when disks crash) and distributed management (so there is no single point of failure). Colossus allows BigQuery users to scale to dozens of petabytes of data stored seamlessly, without paying the penalty of attaching much more expensive compute resources as in traditional data warehouses

In between storage and compute is ‘shuffle’, which takes advantage of Google’s Jupiter network to move data extremely rapidly from one place to another

The mixers and slots are all run by Borg, which allocates hardware resources.

Note: If you are not a person who wants to read long written stuff, then we have a wonderful video for you related to BigQuery Architecture. So bring your popcorn=ns and binge it!!

Advised Optimizations in BigQuery

Controlling Costs

Avoid SELECT (full scan), select only columns needed

(SELECT EXCEPT) -Sample data using preview options for free — Preview queries to estimate costs (dryrun)

Use max bytes billed to limit query costs

Don’t use LIMIT clause to limit costs (still full scan) — Monitor coets using dashboards and audit log

Partition data by date — Break query results into stages

Use default table expiration to delete unneeded data — Use streaming inserts wisely

Bet hard limit on bytes (members) processed per day

Input Data/Data Sources

Avoid SELECT *

Prune partitioned queries (for time-partitioned table, use PARTITIONTIME pseudo column to filter partitions)

Denormalize data (use nested and repeated fields)

Use external data sources appropriately

Avoid excessive wildcard tables

SQL Anti-Patterns

Avoid self-joins, use window functions (perform calculations across many table rows related to current row)

Partition/Skew: avoid unequally sized partitions, or when a value occurs more often than any other value

Cross-Join: avoid joins that generate more outputs than inputs (pre-aggregate data or use window function) Update/Insert Single Row/Column: avoid point-specific DML, instead batch updates and inserts

Managing Query Outputs

Avoid repeated joins and using the same subqueries

Writing large sets has performance/cost impacts. Use filters or LIMIT clause. 128MB limit for cached results

Use LIMIT clause for large sorts (Resources Exceeded)

Optimizing Query Computation

Avoid repeatedly transforming data via SQL queries Avoid JavaScript user-defined functions

Use approximate aggregation functions (approx count)

Order query operations to maximize performance. Use ORDER BY only in the outermost query, push complex operations to the end of the query.

For queries that join data from multiple tables, optimize join patterns. Start with the largest table

HandsOn Exercise of some Optimization Concepts

Exploring Stack Overflow dataset

We would be using the Stack Overflow dataset, it contains the information about posts, tags, badges, comments, users, and more. For browsing the Stack Overflow dataset in the BigQuery User Interface, do follow these steps:

Open the Stack Overflow dataset. The BigQuery User Interface opens in the GCP Console and displays information about the Stackoverflow dataset. In the navigation panel , select the bigquery-public-data. The menu expands to list public datasets. Each dataset comprises one or more tables. Scroll down and select stackoverflow. The menu expands to list the tables in the Stack Overflow dataset. Select badges to see the schema for the badges table. Note the names of the fields in the table. Above the Field names, click Preview to see sample data for the badges table

Query the Stackoverflow dataset

For running a query, follow the underlined steps:

1. Select the Compose new query, present near to the top right of the GCP console.

2. In the Query editor text area, jot down the following query:

SELECT EXTRACT(YEAR FROM creation_date) AS creation_year, COUNT(*) AS total_posts FROM `bigquery-public-data.stackoverflow.posts_questions` GROUP BY creation_year <br> ORDER BY total_posts DESC LIMIT 10

3. Now, select the Run button.

Result would be the number of Stack Overflow posts or questions posted every year.

Create a New Dataset

For creating and loading the table data into BigQuery, first of all create a BigQuery dataset for holding the data by following underlying steps:

1. Select the name of your project as part of the setup, In the GCP console navigation panel.

2. Select the Create dataset present on the right side, in the details panel.

3. When the Create dataset dialog appears for Dataset ID type stackoverflow. Then leave all of the other default settings in place and click OK.

Create a New Table with 2018 StackOverflow Posts

Now that we are done with the creation of a BigQuery dataset, we can create a new table in this dataset. So, for creating a table with data inherited from the existing table, we would query the 2018 Stack Overflow posts dataset and then we would write the results to a newly created table, by following underlying steps:

1. Near the top right of the GCP console, select Compose new query.

2. In the Query editor text area, jot down the following SQL query for creating a new table.

CREATE OR REPLACE TABLE `stackoverflow.questions_2018` AS SELECT id, title, accepted_answer_id, creation_date, answer_count , comment_count , favorite_count, view_count, tags FROM `bigquery-public-data.stackoverflow.posts_questions` WHERE creation_date BETWEEN '2018-01-01' AND '2019-01-01';

3. Select the Run button.

Query the New Table with 2018 Stack Overflow Posts

Now that you have created a BigQuery table, let’s run a query to return Stack Overflow posts with questions and titles along with a few other statistics such as number of answers, comments, views and favorites. Complete the following steps:

1. Near the top right of the GCP console, select Compose new query.

2. In the Query editor text area, jot down the following SQL query

SELECT id, title, accepted_answer_id, creation_date, answer_count , comment_count , favorite_count, view_count FROM `stackoverflow.questions_2018` WHERE creation_date BETWEEN '2018-01-01' AND '2018-02-01' AND tags = 'android';

3. Select Run. The query would return the Stack Overflow questions that have been created in the month of January 2018 that are tagged as android.

4. BigQuery caches the query results, by default. Now run the exact same query and we would observe that BigQuery took way less time for returning the results because it returns the results from the cache.

5. Again, run the same query again but this time just disable the BigQuery caching.

Note: We would disable the cache for the rest of the exercise to be actually fair in performance comparison against the partitioned and clustered tables that we would be executing ahead.

For disabling the Cache Query, In the query editor click on the More button and select the Query settings.

6. Under Cache preference, uncheck Use cached results.

7. In the query results, we would observe the time it took for the query to complete and amount of data processed to get the results.

Create a New Partitioned Table with 2018 StackOverflow Posts

For creating a partitioned table with data inherited from existing table or query, we would query the 2018 Stackoverflow posts dataset and then we would write results to a new table.

1. Select Compose new query, which is present near the top right of the GCP console

2. In the Query editor text area, jot down the following SQL query for creating a new table.

CREATE OR REPLACE TABLE `stackoverflow.questions_2018_partitioned` PARTITION BY DATE(creation_date) AS SELECT id, title, accepted_answer_id, creation_date, answer_count , comment_count , favorite_count, view_count, tags FROM `bigquery-public-data.stackoverflow.posts_questions` WHERE creation_date BETWEEN '2018-01-01' AND '2019-01-01';

3. Finally, select Run.

This query would help to create a new table named as questions_2018_partitioned in the stackoverflow dataset with the data resulting from running a query on the BigQuery StackOverflow dataset bigquery-public-data.stackoverflow.posts_questions

Query the Partitioned Table with 2018 Stack Overflow Posts

Now that we are done with creating a BigQuery partitioned table, let’s run the exact same query, but this time the only difference would be that it is on the partitioned table, for returning the StackOverflow posts with questions and titles along with a few other statistics such as number of answers, comments, views and favorites. Complete the following steps:

1. Near the top right of the GCP console, select Compose new query.

2. In the Query editor text area, jot down the following SQL query

SELECT id, title, accepted_answer_id, creation_date, answer_count , comment_count , favorite_count, view_count FROM `stackoverflow.questions_2018_partitioned` WHERE creation_date BETWEEN '2018-01-01' AND '2018-02-01' AND tags = 'android';

3. Select Run with BigQuery caching disabled.

4. In the query results, we would observe that the time it took for the query to complete and volume of data processed to get the results.

Create a New Clustered Table with 2018 Stack Overflow Posts

In this section, we would be creating a new table partitioned on creation_date and clustered on the tags column based on the query access pattern. For creating a clustered table with data from an existing table or query, we would be querying the 2018 Stack Overflow posts table and write down the results to a new table.

1. Near the top right of the GCP console, select Compose new query.

2. In the Query editor text area, jot down the following SQL query for creating a new table

CREATE OR REPLACE TABLE `stackoverflow.questions_2018_clustered` PARTITION BY DATE(creation_date) CLUSTER BY tags AS SELECT id, title, accepted_answer_id, creation_date, answer_count , comment_count , favorite_count, view_count, tags FROM `bigquery-public-data.stackoverflow.posts_questions` WHERE creation_date BETWEEN '2018-01-01' AND '2019-01-01';

3. Select Run.

Query the Clustered Table with 2018 Stack Overflow Posts

Now that we are done creating a BigQuery clustered table, let’s run the exact same query again, but this time on the partitioned and clustered table, for returning StackOverflow posts with questions and titles along with a few other statistics such as number of answers, comments, views and favorites

1. Near the top right of the GCP console, select Compose new query.

2. In the Query editor text area, jot down the following SQL query

SELECT id, title, accepted_answer_id, creation_date, answer_count , comment_count , favorite_count, view_count FROM `stackoverflow.questions_2018_clustered` WHERE creation_date BETWEEN '2018-01-01' AND '2018-02-01' AND tags = 'android';

3. Select Run with BigQuery caching disabled.

4. In the query results, we would observe the time it took for the query to complete and amount of data processed to get the results.

Conclusion

BigQuery is a serverless, highly scalable, and cost-effective data warehouse which is provided by Google. Using this one can analyze TBs of data in seconds but some optimization can make it to Milliseconds such as Query Caching, Partitioning, Clustering, avoiding SELECT *, and many more.

Additional Resources