Build Robust Google BigQuery Pipelines with Python: Part III
By: Chubing Tripepi
Tags: Google Sheet, Data Pipeline, Bigquery Python Api, Google Cloud Platform, Bigquery

At Spotify, Google Sheet is a very popular office tool. One of the advantages of BigQuery is its capability to interact with data stored in Google Sheet. You can access, analyze, visualize, and share billions of rows of data from your spreadsheet with Connected Sheets, the new BigQuery data connector. This can be very handy when Data Scientists or engineers are collaborating with users who are less technical. However, there are some unexpected behaviors in BigQuery when it comes to Google Sheet, and they might not seem intuitive at first. This article intends to dive deep into different use cases of Google Sheet in BigQuery and help avoid confusion due to those unexpected behaviors, especially in the Python pipeline environment.

I. Create Table from Google Sheet to BigQuery

The phrase “create table” is misleading in this case — a more accurate term would be “create table connection.” BigQuery is establishing an active connection rather than actually importing a dataset into BigQuery. But BigQuery console uses this term, so we will stick with it for now.

Here is an overview of what you need to do to create table from Google Sheet:

Select “Drive” for Create table from Copy and paste Sheet URI to the next box Select Google Sheet for the File format Write down the metadata for the table such as project, dataset, and table names; define schema for the table — I recommend manual input for the schema as the auto detect don’t always work right Define Header rows to skip , 0 if there’s no header, n if you have n header rows to skip

Create Table from Google Sheet

Once the table is created, you will be able to query it in BigQuery. However, one common mistake is thinking that the data is now stored as a separate copy in BigQuery. That is not the case. What BigQuery does here is merely creating an active connection to the Google Sheet, and here are the implications:

If you place any filter on the Google Sheet, BigQuery will not be able to display the entire dataset; filters in Google Sheet actually are reflected immediately and affect how the data look like in BigQuery. So always use filter for individual views if you want to access all the data in BigQuery. If there is any schema change, such as column name change, column type change, you cannot edit the existing schema in BigQuery. Delete the current BigQuery table and redo the Create table process.

II. Read BigQuery Google Sheet Data in Python

If you are trying to use a service account to run the job, make sure that you add the service account as an editor for the Google Sheet.

The most important step to set up reading a Google Sheet as a BigQuery table is to modify the scope for BigQuery Client in the Python BigQuery API.

import pandas as pd

from google.cloud import bigquery scopes = (

'https://www.googleapis.com/auth/drive',

'https://www.googleapis.com/auth/drive.file',

'https://www.googleapis.com/auth/spreadsheets',

)

bigquery.Client.SCOPE += scopes client = bigquery.Client()

And then, here is a one line code to convert your query result to a pandas dataframe .

gs = client.query("""SELECT * FROM `YOUR_PROJECT.YOUR_DATASET.{table}`""".format(table=table_name)).result().to_dataframe()

III. Write to Google Sheet in Python

There are several packages that will allow you to write to Google Sheet in Python. I will use pygsheets as an example. The package is pretty straightforward if you don’t plan to run the job in a pipeline. However, there is a workaround needed if you plan to use the package in a pipeline; for security reasons we cannot store any service account key files in our cloud server, so the regular function will not work. Let’s see some examples in these two scenarios.

Assume that you plan to use a service account key for this job. Here is a quick intro to service account.

A service account is an account associated with an email. The account is authenticated with a pair of public/private keys, making it more secure than other options — as long as the private key stays private. — Pysheets

To create a service account, follow these steps:

Go to “Credentials” tab and choose “Create Credentials > Service Account Key”. Next choose the service account as ‘App Engine default’ and Key type as JSON and click create:

3. Next click on the download button to download the client_secret[…].json file; make sure to remember where you saved it as a local json file:

Now that we have the service account key file ready, let’s import the required packages:

import requests

import json

import pygsheets

import styx_secrets

from google.oauth2 import service_account

from google.cloud import bigquery

import pandas as pd

import numpy as np

The authorization process is simple:

gc = pygsheets.authorize(client_secret='path/to/client_secret[...].json')

However, if you have a pipeline job, the process is slightly more complicated, as we need to use a dictionary as key as our workaround.

First we need to encrypt our service account key. At Spotify, we have this package called styx-secrets for encryption purposes. Note that decode for utf-8 is needed in order for pygsheets to recognize this key.

import styx_secrets key1 = styx_secrets.decrypt("FIRST_HALF_OF_ENCRYPTED_KEY_STRING").decode("utf-8")

key2 = styx_secrets.decrypt("SECOND_HALF_OF_ENCRYPTED_KEY_STRING").decode("utf-8")

I had to break down the key into two strings because the original key string was too long, and styx-secrets won’t take it.

And then we can prepare the key file in a dictionary , this is basically a copy and paste of the json key file, except swapping the real key with your encrypted string (note that there are some formatting changes needed again after decryption):

sc_key = {

"type": "service_account",

"project_id": "YOUR_PROJECT",

"private_key_id": "YOUR_PRIVATE_KEY_ID",

"private_key": ("-----BEGIN PRIVATE KEY-----" + key1 + key2 + "-----END PRIVATE KEY-----

").replace("\

", "

"),

"client_email": "YOUR_SERVICE ACCOUNT_EMAIL",

"client_id": "SERVICE_ACCOUNT_CLIENT_ID",

"auth_uri": "https://accounts.google.com/o/oauth2/auth",

"token_uri": "https://oauth2.googleapis.com/token",

"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",

"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxxxxxx"

} credentials = service_account.Credentials.from_service_account_info(

sc_key,

scopes=[

'https://www.googleapis.com/auth/spreadsheets',

'https://www.googleapis.com/auth/drive',

'https://www.googleapis.com/auth/drive.file', ]) gc = pygsheets.authorize(custom_credentials=credentials)

Now that we are done with authorization, we can have some fun! To write to a Google Sheet, here is an example.

Suppose you have a Google Sheet tracking incidents for your pipeline so your team can easily analyze the data:

# open the google spreadsheet

sh = gc.open('Data Incident Tracker')



# select the first sheet

wks = sh[0] # import previous events

track_prev = client.query("""SELECT * FROM `YOUR_PROJECT.YOUR_DATASET.{table}`""".format(table=table_name)).result().to_dataframe()



# current event

track_now = [{

'incident_date': date_param,

'num_of_tables': len(missing_list),

'table_list': ', '.join(missing_list),

'alert': True,

'root_cause': root_cause}, ]



# create new data frame with all incidents



tracker_df = pd.concat([track_prev, pd.DataFrame(track_now)])



# update the first sheet with tracker_df



wks.set_dataframe(tracker_df.replace(np.nan, '', regex=True), 'A1')

Lastly, please note that if you have any None value in your table, they will actually be uploaded as a “NaN” string in Google Sheet. To avoid that, replace them with empty strings using replace(np.nan, ‘’, regex=True) and the cell will be uploaded as an actual Null in the Google Sheet.

This is the last article of the Google BigQuery in Python series. I hope you find them useful. Please feel free to reach out or comment if you have any questions. Cheers!