A gentle introduction to the 5 Google Cloud BigQuery APIs
By: Antonio Cachuan
Tags: Python, Bigquery, Data Engineering, Data Science, Towards Data Science

1. BigQuery API

The principal API for core interaction. Using this API you can interact with core resources as datasets, views, jobs, and routines. Up today exists 7 client libraries: C#, Go, Java, Node.js, PHP, Python, and Ruby.

Example

For this example, I will use the python client library for the BigQuery API on my personal computer. Consider that you need to have python already installed.

Installing Python

As a recommendation install Visual Studio Code and enter the terminal. I’ll use it for all the examples.

1.1 Install the client library

pip install --upgrade google-cloud-bigquery

Installing the client library

1.2 Setting up authentication

To access the BigQuery service first you need to create a service account and setting an environment variable.

A service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs [GCP Doc].

Enter the Google Cloud Console and then APIs & Services > Credentials > + Create Credentials > Service Account

Creatin a service account

Creatin a service account

After the SA is created we need to set up a JSON key.

Automatically a .json file will be downloaded, keep it safe since it is your key to your BigQuery resources.

Back to the Visual Studio Code terminal provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS

set GOOGLE_APPLICATION_CREDENTIALS=D:\medium\bigquery-apis\key\key_bqsa.json

For this example, I’ll query the Covid dataset.

You could also get the result in a pandas dataframe, fir install the following libraries

pip install pandas

pip install pyarrow

pip install google-cloud-bigquery-storage

2. BigQuery Data Transfer API

API used for ingestion workflows. If you want to include periodic ingestion from Google Cloud Storage or get analytics data from other Google Service like Search Ads 360, Campaign Manager or YouTube, or other third parties services like Amazon S3, Teradata, or Amazon Redshift.

2.1 First install the library

pip install --upgrade google-cloud-bigquery-datatransfer

2.2 Enable the API

Important: You need to enable billing https://console.developers.google.com/billing

Let’s move the data from a YouTube Channel to BigQuery. Go to Marketplace and look for YouTube Channel Transfers and click on Enroll.

Click on Configure Transfer. Lets set some inputs like schedule options and destination setting.

This will prompt a windows asking to access your YouTube data.

When you scheduled a query this API is used.

Now let’s start using this API. First showing all the Data Transfer created

The output give us the Transfer ID.

With the Transfer ID, using the BigQuery Data Transfer API I’m able to run it.

Executing the code returns the job values.

In the BigQuery interface you could follow the execution. In this case I didn’t have YouTube data so this trigger an error.

3. BigQuery Storage API

This API exposes high throughput data reading for consumers who need to scan large volumes of managed data from their own applications and tools [Google Doc].

Let´s build a simple example

3.1 Install the client library

pip install --upgrade google-cloud-bigquery-storage

3.2 Set up the authentication

Follow the steps from 1 to create a service account and get the JSON with the service account key.

3.3 Example 1

Getting the top 20 page views from Wikipedia. This example use both APIs and run a custom query.

3.4 Example 2

If we need fine-grained control over filter and parallelism the BigQuery Storage API read session could be used instead of a query.

The following code only changes to use the Apache Arrow data format.

4. BigQuery Connection API

This API is for establishing a remote connection to allow BigQuery to interact with remote data sources like Cloud SQL. This means BigQuery is enabled to query data residing in Cloud SQL without moving the data.

Let´s build a simple example

4.1 Install the client library and enable the API

pip install --upgrade google-cloud-bigquery-connection

Enable the API accessing the console.

4.2 Set up the authentication

Follow the steps from 1 to create a service account and get the JSON with the service account key.

4.3 Add an external source

For this example, I’ve deployed a Cloud SQL instance in the same project. Next, I’ve added a simple MySQL database as an external source in the BigQuery interface.

Adding the connections parameters

Running the federated query

4.4 Example

This example uses the API to list the connections. For more examples check the official repository.

5. BigQuery Reservation API

This API allows provisioning and manage dedicated resources like slots (virtual CPU used by BigQuery to execute SQL queries) and BigQuery BI Engine ( fast, in-memory analysis service) memory allocation.

5.1 Install the client library

pip install --upgrade google-cloud-bigquery-reservation

5.2 Set up the authentication

Follow the steps from 1 to create a service account and get the JSON with the service account key.

5.3 Enable Reservation API

Go to the BigQuery UI and click on Reservations

5.4 Buy Slots

This feature is more recommendable to organizations that desire to have predictable pricing (flat-rate). Remember that by default on BigQuery you pay per-query (on-demand). The next images simulate a Slot reservation process in order to use later the API.

5.5 Example

Use the API to list the reservations. For more actions check the repository.

Conclusions

BigQuery gives us many features and with the help of these APIs, we can extend the functionality. All the code is available on Github.

PS if you have any questions, or would like something clarified, ping me on Twitter or LinkedIn I like having a data conversation 😊

Useful Links

Core API

Transfer API

Storage API

Connection API

Reservation API