Connecting Google Sheets → BigQuery → Google Data Studio
By: Sam Thurman
Tags: Data Engineering, Bigquery, Google, Google Data Studio, Data Analysis

Connecting Google Sheets → BigQuery → Google Data Studio

Overview

The structure of this article is as follows:

Introduction

Advantages of this workflow

Steps we’ll follow

The Process

If you’re here strictly for the tutorial, navigate down to the Steps we’ll follow section.

Introduction

The “Google Stack” is becoming exceedingly popular amongst non-technical teams who need to share and analyze their data. Today we’ll learn how to leverage BigQuery and Google Data Studio to store, transform, visualize, and share our data in a cloud environment.

The process of piping data through the Google atmosphere can be a pain to get up and running. Once implemented, however, the benefits to non-technical teams are substantial.

Advantages of this workflow:

Google Sheets

Collaboration — Everyone has access to the most up-to-date version of the data. Data does not have an “owner”, no “pushing to GitHub” required for team members to have access to changes.

Scalability — Perfect for up to 100k records.

Linking to other apps — Whether you want to add a visual to a Google Slide or connect your sheet to BigQuery, there's an endless supply of Google Connectors that makes connecting your data to other applications easy.

Version control — Google Sheets stores versions of your google sheet for reference (similar to a commit log). Did you make an unintended edit? No worries, you view previous versions and reset your sheet under “See Version History”.

Linking between sheets — Google’s cloud functionality makes it exceedingly easy to connect your sheet with other sheets/workbooks and keep them connected. Paths are stored by Google, so renaming/moving files won’t affect your connections between files.

Plugins — Huge library of plugins for visualization, number-crunching, and connecting to external data sources.

Connect to an external data source — The never-ending supply of Connectors and plugins makes it easy to link your sheet to an external data source.

BigQuery

Cost-effective, highly-scalable, NoOps — Pay-as-you-go, costs based on usage. Automated scaling, no infrastructure to manage.

Streamlined ETL and BI — Endless options of Connectors make it extremely easy/fast to set up a data warehouse, make some transformations, and analyze your data.

Create and test ML models using SQL — BigQuery’s BigQuery ML feature allows users to create, run, and test ML models using SQL queries.

Google-managed service, no maintenance —BigQuery updates are delivered and implemented on your systems automatically. No infrastructure to manage on your end.

Built-in DR and data protection — %99.9 SLA guaranteeing uptime, some of the strongest security and data-governance of any cloud-based warehousing service.

Google Data Studio

Free to use

Highly interactive — Interactive component design makes it easy (and sometimes fun) to explore your data.

Near unlimited widget options — No limit on widgets, add as many components to your report as you need. Large catalog of user-created widgets allows for untempered creativity.

Pull data from up to 12 sources — Support for connecting 12 unique data sources with options for blending data from data sources.

Sharable — Of course, Google makes it seamlessly easy to deploy and/or share reports with teammates/stakeholders/anyone with the URL.

Easy to use — Friendly interface for non-technical parties.

Dynamic reporting — Real-time updates to data and application, whenever something is changed, the application is updated.

Steps We’ll Follow

These steps are meant to be followed in order. If you’ve already completed some of the steps, feel free to skip ahead to the most relevant section.

Log into BigQuery and navigate to Cloud Console

Create a new project

Enable BigQuery Connector API

Create a new dataset

Connect your Google Sheet as a table

Log into Google Data Studio

Create a new data source, connect our data using BigQuery connector

Use data in a report

The Process

*Briefly on BigQuery data organization:*

Tables are the lowest-level organizer in a BigQuery project, holding rows and columns of data (we will be creating a table from our Google Sheet). Datasets hold tables and are the mid-level organizer. Datasets are stored with the resources for your project (the highest-level organizer).

project-name/

dataset-for-project/

table1/ (this will be our Google Sheet, could be anything)

...

table2/

...

Log into BigQuery and navigate to Cloud Console

We start by logging into the Google Cloud platform and navigating to the Console .

Create a new project

In the upper left hand corner, click My First Project (our current/default project. This will bring up the project selection window.

Click NEW PROJECT to create a new project for our data.

Give our project a name (something unique, preferably short) and press CREATE .

You should now see the name of your new project displayed in the upper left-hand corner, indicating we are now viewing resources for this project.

Enable BigQuery Connection API

We have to enable the BigQuery Connection API for every new project. To search for and add a resource/connector, we can use the search field at the top of our dashboard.

Search for BigQuery and select it from the dropdown. This will open the BigQuery editor.

On the left-hand side of the screen, you’ll see a navbar that allows you to access your saved queries, jobs, etc. At the bottom of this section, underneath the search field, you’ll see your current project listed as a resource. Any new data sources for this project will be stored in this resource section.

To easily enable BigQuery for your project, select ADD DATA + and then select External data source .

You’ll be given this prompt asking if you wish to enable the BigQuery Connection API. Click ENABLE API .

Create a new dataset

Now that we have our BigQuery Connection enabled, it’s finally time to add some data to our project.

Select your project name from the Resources list.

Click CREATE DATASET . This will bring up the dataset creation panel.

Set a name for your dataset. Unless you have a good reason, leave the other two options set to “Never” and “Google-managed key”. When finished, click Create dataset .

You’ll now see your new dataset appear in the resources tab underneath your project name, indicating your new dataset belongs to that project.

Connect your Google Sheet as a table

Okay, now it’s really time to add our data. I promise.

Select your new dataset from the resources tab and click CREATE TABLE .

This will bring up the table creation panel.

To add a Google Sheet from Drive as a table, navigate and open your file in Drive and copy the URL.

Next, back in the table creation panel, select “Drive” from the Create table from: dropdown.

Paste your Google Sheet’s URL into the provided field.

Under File format: , select “CSV”, not “Google Sheet”. This tells BigQuery that column names for this table can be found in the first row of the file (typical for most Google Sheets, standard format for CSV files).

Next, we have to select a location to store this table. Select your newly created project and dataset names from the corresponding dropdowns.

Enter a name for your new table. This table will live inside your new dataset, inside your new project.

Finally, click Create table .

You’ll now see your new table appear under your new dataset inside the resources for your project.

Log into Google Data Studio

Now that we have our data available through BigQuery, it’s time to put it to use.

Navigate to your Data Studio landing page.

Create a new data source, connect our data using BigQuery connector

In order to use data from BigQuery, we have to create a new BigQuery Connector in Data Studio. In the upper left-hand corner, click Create and select Data Source from the dropdown.

This will open up our data source creation panel. Search for/select the BigQuery connector.

This will open up the creation options for the connector. Select your new project name from the MY PROJECTS tab, then select the corresponding dataset and table you wish to create a connection for. When you’re done, click CONNECT in the upper left-hand corner. This may take a second.

When that process is done running, you’ll be presented with this summary page detailing various aspects of your data source. From here, you have the option to directly create a report and add this data to it (yellow arrow), but let’s do it the long way so we’re familiar with the process. Navigate back to your Data Studio home (click on the little sliders next to your BigQuery connector name.

Use data in a report

Let’s create a new Report to visualize our data.

First, give this report a name, then, select Add data .

This will bring up a navigation panel where we can easily browse and connect to any of the data we have stored as BigQuery connectors.

For now, navigate to My data sources , select our new BigQuery connector, and click Add .

You’ll be given this prompt telling you that you’re about to add some data, click ADD TO REPORT .

By default when adding a new data source, a Table component will be created for you representing some form of record count. You’ll see your Bigquery connector appear under the Data sources for your chart, and you’ll be able to see a list of the available fields (columns) from your data source. You can now use this BigQuery connector as a data source for any new components you wish to add.

Fin

And that’s it! I know that was a lot of steps, and probably more detail than you needed, but think about what we’re able to do now… You’ve practically architected an entire modern DevOps flow 😉 (kind of)

We now know how to take a raw Google Sheet (or CSV uploaded to Google Drive), store it in a warehouse using BigQuery, and pipe it into a data visualization suite all in the cloud without writing any code!

For a full-time analyst who spends 30+ hours a week building dashboards for a large corporation, this might not be a particularly technically rigorous process, we didn’t even write any SQL! But for non-technical teams looking to take charge of their data analysis, what we’ve just done is a huge win and worthy of applause👏👏👏