Google Sheets, Meet Pandas DataFrame | by Nicholas Ballard | Towards Data Science

Maybe it’s from too many hours behind the keyboard. But sometimes my reading and watching on the internet gives me the feeling I’m one of an endangered species of developer. One who’s working inside spreadsheets. Every single day.
Can you believe it? Developers, going about their lives like spreadsheets don’t exist. If you are one of these people, I’d like to meet you. To know that you exist, and this isn’t all a paranoid rant raging between my ears.
For the rest of us, spreadsheets are very much a thing. Love them or hate them, those binary rows and columns are going nowhere. To cut the philosophy lesson short: spreadsheets are. Every non-developer employee save the cleaning staff lives inside spreadsheets.
“Spreadsheets are.” Submit this existential bombshell to a university; an honorary doctorate in philosophy is sure to follow.
Excel and Google Sheets, they’re going nowhere; neither is our productivity until we put automation between us and the knowledge worker ritual of passing the spreadsheet.
Personally, I love a good spreadsheet. For munging data, quick and dirty calculations, visualizing 1-dimensional times series... 4 out of 5 times I will open Excel or Google Sheets, rather than fire up a Jupyter server.
Except, spreadsheetitis in a business is a symptom of not knowing how to move data. I have personally seen the “human router” problem so bad where more than 20 (twenty!) people touched a version of a spreadsheet before it was finally–mercifully–put to rest in “the database” (another spreadsheet).
And THE Spreadsheet. You know the business is in data hell when there’s The Spreadsheet, with a capital “S”. The one managers think is the business.
There’s a few libraries we will talk about using for Google Sheets I/O in Python. No matter the approach, you have to enable the Google Sheets API inside the maze-like console for Google Cloud Platform (Google’s version of AWS and Azure). Fortunately Google–go figure–has a great search feature in the console that we’ll use to get you set up.
Go to console.cloud.google.com/, sign in with the Google account with access to the spreadsheet you want to automate. The API is available for both GSuite and free Google accounts. If it’s a work-owned spreadsheet, the admin with GCP access is going to have to follow these steps and give you the credentials file.
Select the project in the navbar on the top. Every resource is kept inside a project, like resource groups in Azure. If you do not have one yet, select the project dropdown and “NEW PROJECT”. Then search for “Google Sheets API”, which if you are a sadist and want to explore the dashboard by mouse, is under “APIs & Services”.
“Enable” the API.
Now you need to do the same for Google Drive API.
Next up, you need to create a service account, which you will then create credentials for.
Steps:
You are in Google Cloud Platform > Google Sheets API at console.cloud.google.com/apis/api/sheets.googleapis.com
Go to credentials 🗝 Credentials on the sidebar.
Select + CREATE CREDENTIALS > Service Account. OAuth is a different story. This article covers a server-side, in-house use. Create the service account.
Next view is to select an IAM role for the service account. This is optional, skip it.
Next up is “Grant users access to this service account (optional)”. Also skip.
Now you are back on the “Service accounts for project [PROJECT NAME]” screen. Select the service account in the table, either clicking on the hyperlinked Email field, or hamburger menu (three dots) > Create key.
Either way, ADD KEY or Create key, select JSON key type and CREATE. This downloads a JSON file to your computer.
Last thing: the spreadsheet. Here I created one for this example. Note the highlighted section of the URL in the Omnibar. That is the document’s ID. Most libraries give the choice of referencing the whole URL or just this ID.
You can take a look at and download the spreadsheet from here.
Open the credentials.json file you downloaded from GCP for the service account. Copy the email value in the client_email property. Share the Google Sheet with this email. This gives the service account access to the spreadsheet, which was created on another account (not the service account).
Phew. That’s all the Google infrastructure stuff. I promise.
Now we can get down to business writing Python code.
Google has an API for Sheets, like it does for all Google Drive products. There is nothing you cannot do with its Python library.
Still you would be mad to use it.
My hard line on spreadsheets is they are a portable, visual way to deliver data to business users. A rectangle of data, cell A1 on the top left. Not to play Bob Ross🎨🖌 with conditional formatting, islands of data in the same worksheet forming archipelagos, or creating a Tableau-like dashboard of charts.
Not to say all those things aren’t great. I’ve went down those rabbit holes with spreadsheet automation many times. It’s just, the payoff-to-time-spent-coding ratio falls off a cliff pretty fast when you dive into all the features Google Sheets built in over the years.
This article is about data in, data out. Fastest time to solution wins.
On the Python data side, that means using pandas DataFrames. On the Google Sheets connector side, it’s more up in the air. Time to go shopping on GitHub.
For that, here are three of my favorite third party Python libraries wrapping the Google Sheets API:
EZSheets
Al Sweigart can do no wrong. His book Automate the Boring Stuff With Python singlehandedly taught me how to code for money. I still reference it a few times a month.
Sweigart is this giant brain with a talent for taking the complex and breaking it down into baby food I can digest. EZSheets (GitHub) is his creation, along with sibling interfaces for Gmail, Calendar, and Drive.
EZSheets shines with its clear methods for updating a selection of data all at once. Also, for selecting columns and rows. I haven’t found another library that captures the complexity of iterating over cells quite like this one. Other libraries are no better than the Sheets API, where the nested for loops for processing data give me vertigo.
gspread
gspread (GitHub) is a library by another brain, Anton Burnashev. gspread is popular to the point of being the de facto library for Python developers wanting to work with Google Sheets.
And with good reason. The documentation is great, the source code is clear with a sensible layout and good docstrings. Plus, gspread’s interface wraps some Google Sheets formatting functionality. It also has methods for working with pandas and NumPy arrays.
gspread-dataframe (Winner)
A wrapper for the gspread library built by Robin Thomas, gspread-dataframe (GitHub) is my go-to package for reading and writing Google Sheets with DataFrames.
The documentation is good, but somewhat unnecessary. The source code is brief, with two functions, get_as_dataframe and set_with_dataframe.
gspread-dataframe is what we will be using for this tutorial.
Scenario: We have a table of data about for an ongoing movie night. Coworkers need to add values to a column manually on a daily basis. As we schedule new movie nights, we need to collect the handwritten data entered in the sheet, as well as add new movies to the top.
This example is fiction, but I have had multiple clients that were medium size tech companies request just this feature. You can imagine there is a database with tables like movies and movienights, and there's a view function joining those into this one table that the coordinators of the movie night help fill out.
Short of a full-blown web app or form software, using a spreadsheet as the presentation and data entry layers is actually quite convenient.
Bonus: With pandas 1.1.0, we can generate a markdown table without index from a method on the DataFrame!
df.head(1).to_markdown(index=False)​‘| IMDB URL | IMDB Score | Length | Attendees | Mood | Staff Notes |\n|: — — — — — — — — — — — — — — — — — — — — — — — — — — — | — — — — — — -:| — — — — -:| — — — — — — :|: — — — -|: — — — — — — — — — — -|\n| https://m.imdb.com/title/tt5013056/?ref_=m_ttls_tt_20 | 7.9 | 106 | 18 | 😓 | Bane flies a plane ✈ |’
Our data:
First install gspread_dataframe. Since it is a wrapper for gspread, we will need that library and its oauth2client dependency.
pip install gspread_dataframe gspread oauth2client
Check you can connect to the sheet:
$ ipythonPython 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]Type 'copyright', 'credits' or 'license' for more informationIPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.In [1]: import gspread   ...: from gspread_dataframe import (get_as_dataframe,   ...:                                set_with_dataframe)   ...: from oauth2client.service_account import ServiceAccountCredentials   ...: scope = ["https://spreadsheets.google.com/feeds",   ...:          "https://www.googleapis.com/auth/drive"]   ...: credentials = ServiceAccountCredentials.from_json_keyfile_name("./credentials.json", scope)   ...: gc = gspread.authorize(credentials)   ...: gc.open_by_key("11zmk55OFpFL5HW_MhPfqfC-GjRQvHlI_M-jtHcBzyzA")Out[1]: <Spreadsheet 'Google Sheets Automation! 🐍' id:11zmk55OFpFL5HW_MhPfqfC-GjRQvHlI_M-jtHcBzyzA>
If you have a problem connecting, read the Exception thrown by gspread, it's pretty informative. Remember, the downloaded keys for the JSON file has a client_email. That email needs the sheet shared with it. The OAuth library needs access to the credentials.json at the right file path. Remember to rename the file to credentials.json.
Now to write to Google Sheets.
# gsheets.py​import gspreadfrom gspread_dataframe import (get_as_dataframe,                               set_with_dataframe)from oauth2client.service_account import ServiceAccountCredentialsimport pandas as pd​​FILE_KEY: str = "11zmk55OFpFL5HW_MhPfqfC-GjRQvHlI_M-jtHcBzyzA"SHEET_NAME: str = "movie night"​​def _get_worksheet(    key:str,    worksheet_name:str,    creds:"filepath to Google account credentials"="credentials.json",    ) -> gspread.Worksheet:    """ return a gspread Worksheet instance for given Google Sheets workbook/worksheet """    scope = ["https://spreadsheets.google.com/feeds",             "https://www.googleapis.com/auth/drive"]    credentials = ServiceAccountCredentials.from_json_keyfile_name(creds, scope)    gc = gspread.authorize(credentials)    wb = gc.open_by_key(key)    sheet = wb.worksheet(worksheet_name)    return sheet​​def write(sheet: gspread.Worksheet, df: pd.DataFrame, **options) -> None:    set_with_dataframe(sheet, df,                     include_index=False,                     resize=True,                     **options)
And using these functions:
sh: gspread.Worksheet = _get_worksheet(FILE_KEY, SHEET_NAME)​""" Here you are getting your data. Usually this would be adatabase. But it could be another spreadsheet, serialized data,generated from Python code... wherever. """df = pd.read_pickle("./movienight2.pickle")​write(sh, df)
Options for set_as_dataframe (credit Robin Thomas, gspread-dataframe.py):
:param worksheet: the gspread worksheet to set with content of DataFrame.:param dataframe: the DataFrame.:param include_index: if True, include the DataFrame's index as an        additional column. Defaults to False.:param include_column_header: if True, add a header row or rows before data with        column names. (If include_index is True, the index's name(s) will be        used as its columns' headers.) Defaults to True.:param resize: if True, changes the worksheet's size to match the shape        of the provided DataFrame. If False, worksheet will only be        resized as necessary to contain the DataFrame contents.        Defaults to False.:param allow_formulas: if True, interprets `=foo` as a formula in        cell values; otherwise all text beginning with `=` is escaped        to avoid its interpretation as a formula. Defaults to True.
Python writes to Google Sheets!
Nice! (I love seeing computers perform drudgery. Means it’s saving me work. 😄) Notice how the resize=True option makes the worksheet fit the DataFrame shape.
Could be better though. Will make a few tweaks to the DataFrame before writing to the sheet. The dates are formatted as datetime. Also, let’s use the power of cell formulas to neaten up the IMDB links.
def _to_string(df: pd.DataFrame, *cols) -> pd.DataFrame:    for col in cols:        if col in df.columns:            df[col] = df[col].astype(str)    return df​def hyperlink_formula(df: pd.DataFrame, textcol: str, linkcol: str) -> pd.DataFrame:    ix: int = df.columns.tolist().index(textcol)    vals: pd.Series = \        '=HYPERLINK("' + df["IMDB URL"] + '", "' + df["Movie"] + '")'    df.insert(ix, textcol+"_temp", vals)    df.drop([textcol, linkcol], axis=1, inplace=True)    df.rename(columns={textcol+"_temp": textcol}, inplace=True)    return df​sh: gspread.Worksheet = _get_worksheet(FILE_KEY, SHEET_NAME)df = pd.read_pickle("./movienight1.pickle")​df = _to_string(df, "Date", "Movie")df = hyperlink_formula(df, "Movie", "IMDB URL")​write(sh, df)
Much cleaner. And now, the Movie column has working hyperlinks!
As new movies nights are inserted into the database, the scheduler running the Python script updates the sheet:
The movie night secretary counts heads, reads the crowd, and writes comments on the event. Things people still do better than machines.
Now for Python to slurp it up. To upsert into a database, save in an S3 bucket, file system, or somewhere.
For this we use the other function provided by gspread_dataframe: get_as_dataframe.
Options for set_as_dataframe (again, credit Robin Thomas, gspread-dataframe.py):
:param worksheet: the worksheet.:param evaluate_formulas: if True, get the value of a cell after        formula evaluation; otherwise get the formula itself if present.        Defaults to False.:param nrows: if present and not None, limits the number of rows read from the worksheet.                Defaults to None. See pandas documentation for more info on this parameter.:param \*\*options: all the options for pandas.io.parsers.TextParser,        according to the version of pandas that is installed.        (Note: TextParser supports only the default 'python' parser engine,        not the C engine.)
The code:
def read(sheet: gspread.Worksheet, **options) -> pd.DataFrame:    return get_as_dataframe(sheet,                     evaluate_formulas=False,                     **options)​df = read(sh)(df[df["Movie"].str.contains("Rambo")]    .values.tolist())
This returns:
[['2020-08-28',  '=HYPERLINK("https://m.imdb.com/title/tt1206885/?ref_=m_ttls_tt_119", "Rambo: Last Blood")',  6.2,  89,  1.0,  '😭😭😭😭',  'WHY AM I THE ONLY ONE HERE???']]
Spreadsheets are a powerful tool. And with amazing wrappers around the Google Sheets API like EZSheets, gspread and gspread_dataframe, a lot of the work managing them with Python is done for you.
Instead of getting in the way of business automation, using spreadsheets as part of the toolchain with Python... it’s saves us a lot of time and unfun work. Best of all, it stops the ritual of “passing the spreadsheet”. It is a wonderful thing!
Here’s the link again to the Google Sheet used in this article.