Python / Pandas & BigQuery in 7 minutes
By: Michael Lisboa
Tags: Google Cloud Platform, Data Science, Pandas, Python, Bigquery

The Django Tango.

I guess the first thing to consider is what, exactly, are we going to do with our data? In this case, let’s imagine we have three models:

We added a ForeignKey in the Thing model because a Member can have many Things. Meanwhile, our Thing model has a ManyToMany relationship to Labels, because a Thing can have many Labels, and a Label can be associated with many Things.

Now, let’s say we want a 12-month comparison of how often a Label is associated to Things that are used by an individual Member to the frequency of use by everybody (all Members). Here’s an example of what that might look like in normal human language:

Over the last year,

We’ve sold a total of 20,000 dresses.

5,000 of those are red dresses.

Cindy has purchased 5 red dresses.

Where a dress is a Thing, red is a Label, and Cindy is a Member. So, we want to see the seasonal frequency of Cindy’s red dress purchases compared to everybody else because we might want to offer Cindy other red things — like shoes, hats, bags — at the right time, in the future.

Get it?

Anyway, let’s start coding.

Go to your Django app views.py and create a new view. I’m calling it push_bigquery() . We’ll start with a query to the Members model:

What we’ve done here is ask for all of our Members and prefetch all Things to get the values() we want, which gives us back a dictionary -like QuerySet.

Then, in the last line, we created a Pandas DataFrame from the values using:

members_df = pd.DataFrame(members_qs)

Now that we have a DataFrame to work with, we’ll do some tweaking to get it into the format we want.

Pick it up in Pandas.

We’re going to do some basic formatting on our DataFrame. Again, we’re keeping this simple, so I’m assuming you know the basics of manipulating data in Pandas.

Add the following to your function:

Here’s a summary of what that does:

First, convert the created_date series to a datetime() object, then

series to a object, then Set created_date as the DataFrame index,

as the DataFrame index, Then get the running cumulative sum of the number of “things”.

The cumulative sum of “labels” associated with “things”.

And finally, the cumulative sum of “things” by each member.

Yeah, I know that sounds really confusing…

We’re doing this because we’re looking to get the frequency of Things and associated Labels members_df[’frequency_of_thing’] by individual Members and comparing that to the frequency of occurrences across all Members.

You’ll probably want to look into optimizing your Pandas queries. I won’t do that here, try Google.

Next we need to format our DataFrame by selecting the columns we want, filtering out any NaN rows and resetting the index:

When we do reset_index we’re saying, “move created_date to a column and set the index rows to 0, 1, 2, 3…”, and do it inplace to update the current DataFrame rather than make a copy, because that’s the way we’re going to insert it into our BigQuery table.

Okay, our members_df is where we want it so we can push it to BigQuery. After all of our hard work getting to this point, you might find this next bit frustratingly simple:

That’s it. Really.

We’re using Pandas to_gbq to send our DataFrame to BigQuery.

'MyDataId.MyDataTable' references the Dataset.table we created earlier.

references the Dataset.table we created earlier. project_id is obviously the ID of your Google Cloud project.

is obviously the ID of your Google Cloud project. if_exists is set to replace the content of the BigQuery table if the table already exists.

Wait, let me talk about “if_exists”.

In this case, if the table already exists in BigQuery, we’re replacing all of the data. You probably wouldn’t want to do that in your real world application.

if_exists has a couple of other methods:

fail , which raises an exception if you try to write to an already existing table, and

, which raises an exception if you try to write to an already existing table, and append which is our preferred option in this case, which will append your data to the existing table.

One other thing: Pickle.

In my case, not only do I want to push all my data to BigQuery, I also wanted a subset of that data for fast lookups across different services. To do this I pickled the last 3 months of my data to disk, using Pandas.to_pickle() :

This makes it easy to read the DataFrame from a shared disk like this:

Extra credit: Running a BigQuery job in Python without Pandas.to_gbq.

If you don’t need to query your database, rather you have it stored as a CSV or another supported format, you can load the file and create a BigQuery job in Python like this:

We could do a bit more optimization on our DataFrame, and maybe run it in a separate thread while doing other work, but this is good enough for the purpose of this article.