BigQuery — Almost All You Need to Know
By: Amulya Rattan Bhatia
Tags: Multi Cloud, Serverless, Google Cloud Platform, Bigquery, Data Warehouse

BigQuery uses remote memory shuffle to gain unprecedented query

speed over big data sets. This enables you to query large amounts of data and get your results very fast. Each query that you execute within BigQuery is split up into a sequence of stages. Each stage is then read from one worker to perform some operation from shuffle and then written back out to shuffle. And then finally when a query is complete within BigQuery, the results are written out to persistent storage and then returned back to the user.

Introduction

In order to use BigQuery, you need to have a project. A dataset is contained within a specific project. Datasets are top-level containers that are used to organize and control access to your tables and views. A table or view must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery. So the hierarchy is: Project →Dataset →Tables/Views.

Jobs are actions that BigQuery runs on your behalf to load data, export data, query data, or copy data. Every job is associated with a specific project that you specify. At a minimum, to run a job, you must have bigquery.jobs.create permissions. The following predefined roles include bigquery.jobs.create permissions:

bigquery.user

bigquery.jobUser

bigquery.admin

Views

A view is a virtual table defined by a SQL query. When you create a view, you query it in the same way you query a table. When a user queries the view, the query results contain data only from the tables and fields specified in the query that defines the view.

Views are treated as table resources in BigQuery so creating a view requires the same permissions as creating a table. At a minimum, to create a view, you must be granted bigquery.tables.create permissions. The following roles include bigquery.tables.create permissions:

bigquery.dataEditor

bigquery.dataOwner

bigquery.admin

You cannot assign access controls directly to tables or views. The lowest level of BigQuery resources to which you are able to grant access is the dataset level.

Giving a view access to a dataset is also known as creating an authorized view in BigQuery. An authorized view allows you to share query results with particular users and groups without giving them access to the underlying tables.

Most individuals (data scientists, business intelligence analysts, data analysts) in an enterprise should be assigned the project-level bigquery.user role.

For your data analysts to query the view, they need to be granted the bigquery.dataViewer role on the dataset containing the view. The bigquery.user role gives your data analysts the permissions required to create query jobs, but they cannot successfully query the view unless they also have at least bigquery.dataViewer access to the dataset that contains the view.

Materialized Views

BigQuery Materialized View are precomputed views that periodically cache results of a query for increased performance and efficiency. BigQuery leverages precomputed results from materialized views and whenever possible reads only delta changes from the base table to compute up-to-date results.

Use cases

BigQuery Materialized View targets the following use cases:

Faster query performance : If you have a raw table and you perform online analytical processing (OLAP) style aggregation that requires significant processing, and you have predictable and repeated queries, such as from extract, transform, load (ETL) or business intelligence (BI) pipelines, you should use BigQuery Materialized View.

: If you have a raw table and you perform online analytical processing (OLAP) style aggregation that requires significant processing, and you have predictable and repeated queries, such as from extract, transform, load (ETL) or business intelligence (BI) pipelines, you should use BigQuery Materialized View. Aggregation of real-time data: If you want access to data for decisions in real-time, BigQuery natively supports streaming capabilities. BigQuery Materialized View integrates with BigQuery streaming and performs aggregations in real-time to provide up-to-date information.

If the base table is partitioned and its materialized view has significant size, then the materialized view should be partitioned as well. As a general guideline, a materialized view has significant size if it is about the same size as, or larger than, one partition of the base table.

Clustering

When you create a clustered table in BigQuery, the table data is automatically organized based on the contents of one or more columns in the table’s schema. The columns you specify are used to colocate related data. Clustering can improve the performance of certain types of queries such as queries that use filter clauses and queries that aggregate data.

When data is written to a clustered table by a query job or a load job, BigQuery sorts the data using the values in the clustering columns. These values are used to organize the data into multiple blocks in BigQuery storage. When you submit a query that contains a clause that filters data based on the clustering columns, BigQuery uses the sorted blocks to eliminate scans of unnecessary data.

Similarly, when you submit a query that aggregates data based on the values in the clustering columns, performance is improved because the sorted blocks colocate rows with similar values. Table clustering is possible for tables partitioned by:

When you use clustering and partitioning together, the data can be partitioned by a date or timestamp column and then clustered on a different set of columns. In this case, data in each partition is clustered based on the values of the clustering columns.

Reservations

BigQuery offers two pricing models for analytics:

On-demand pricing : You pay for the data scanned by your queries. Your cost is based on the number of bytes processed.

You pay for the data scanned by your queries. Your cost is based on the number of bytes processed. Flat-rate pricing: You purchase dedicated query processing capacity.

By default, you are billed according to the on-demand pricing model. Using BigQuery Reservations, you can switch to flat-rate pricing by purchasing commitments. Commitments are purchased in units of BigQuery slots. The cost of all bytes processed is included in the flat-rate price.

BigQuery Reservations introduces three key concepts:

You can purchase dedicated BigQuery slots by procuring commitments.

Optionally, you can split up and isolate your committed BigQuery slots to reservations for workload management purposes.

Use assignments to assign Google Cloud projects, folders, or your entire organization to reservations.

It’s recommended to elect an administration project by only enabling the BigQuery Reservations API on a single project. The administration project creates and maintains ownership of BigQuery Reservations resources (commitments, reservations, and assignments).

Needed BigQuery Role bigquery.resourceAdmin — Allows purchasing and cancelling commitments, modification of reservations, and modification of project-to-reservation assignments.

Controlling costs in BigQuery

Avoid SELECT *

Best practice: Query only the columns that you need.

Sample data using preview options

Best practice: Don’t run queries to explore or preview table data.

Price your queries before running them

Best practice: Before running queries, preview them to estimate costs.

Limit query costs by restricting the number of bytes billed

Best practice: Use the maximum bytes billed setting to limit query costs.

LIMIT doesn’t affect cost

Best practice: Do not use a LIMIT clause as a method of cost control.

View costs using a dashboard and query your audit logs

Best practice: Create a dashboard to view your billing data so you can make adjustments to your BigQuery usage. Also consider streaming your audit logs to BigQuery so you can analyze usage patterns.

Partition data by date

Best practice: Partition your tables by date.

Materialize query results in stages

Best practice: If possible, materialize your query results in stages.

Consider the cost of large result sets

Best practice: If you are writing large query results to a destination table, use the default table expiration time to remove the data when it’s no longer needed.

Use streaming inserts with caution

Best practice: Use streaming inserts only if your data must be immediately available, it costs money while loading data doesn’t.

Optimizing storage in BigQuery

Use the expiration settings to remove unneeded tables and partitions

Best practice: Configure the default table expiration for your datasets, configure the expiration time for your tables, and configure the partition expiration for partitioned tables.

If your tables are partitioned by date, the dataset’s default table expiration applies to the individual partitions.

Take advantage of long-term storage

Best practice: Keep your data in BigQuery.

Rather than exporting your older data to another storage option (such as Cloud Storage), take advantage of BigQuery’s long-term storage pricing.

Querying externally partitioned data

Supported data layouts

The data must follow a default hive partitioned layout. For instance, the following files follow the default layout, with the key/value pairs laid out as directories with an = sign as a separator, and the partition keys are always in the same order.

gs://bucket/table/dt=2019-10-31/lang=en/foo

gs://bucket/table/dt=2018-10-31/lang=fr/bar

The common source URI prefix in this example is gs://bucket/table . The directory structure of a hive partitioned table is assumed to have the same partitioning keys appear in the same order, with a maximum of ten partition keys per table.

More Best Practices

Prefer AVRO file format for loading data into BigQuery.

Prefer ELT into BigQuery over ETL whenever possible.

Migrate your Star or Snowflake schemas as-is to BigQuery and optimize if necessary.

Denormalization can help speed up slow queries. During schema design, use the following principles: