Google Cloud Authentication by Example
By: John Tucker
Tags: Authentication, Google Cloud Platform

The article that I did not think I needed to write but did.

It turns out that understanding how to authenticate to Google Cloud on your workstation is more complicated than one would think.

Please note: This article is not about authenticating a user account to the Google Cloud Console. It is also not about authenticating a service account on a GCE instance.

First, we need to understand that there are two separate components that are authenticated separately: Google Cloud SDK Command Line Tools and Google Cloud Client Libraries.

First, we have Google Cloud SDK Command Line Tools:

The gcloud CLI manages authentication, local configuration, developer workflow, interactions with Google Cloud APIs. With the gcloud command-line tool, it’s easy to perform many common cloud tasks, like creating a Compute Engine VM instance, managing a Google Kubernetes Engine cluster, and deploying an App Engine application, either from the command line or in scripts and other automations.

…

A collection of command-line tools comes packaged with Cloud SDK, including gsutil, bq, and kubectl. The gsutil tool allows you to manage Cloud Storage buckets and objects using the command line. Run queries and manipulate datasets, tables, and entities in BigQuery through the command line with bq. With kubectl, you can deploy and manage Kubernetes container clusters using the command line.

— GCP — Cloud SDK Command Line Tools

And then we have Google Cloud Client Libraries:

Google Cloud Client Libraries are our latest and recommended client libraries for calling Google Cloud APIs. They provide an optimized developer experience by using each supported language’s natural conventions and styles. They also reduce the boilerplate code you have to write because they’re designed to enable you to work with service metaphors in mind, rather than implementation details or service API concepts.

— GCP — Google Cloud Client Libraries

And then we have two different types of accounts that can be authenticated: User and Service Accounts.

User accounts are managed as Google Accounts, and they represent a developer, administrator, or any other person who interacts with Google Cloud. They are intended for scenarios where your application needs to access resources on behalf of a human user. See Authenticating as an end user for more information.

…

Service accounts are managed by IAM, and they represent non-human users. They are intended for scenarios where your application needs to access resources or perform actions on its own, such as running App Engine apps or interacting with Compute Engine instances. See Authenticating as a service account for more information.

— GCP — Authentication Overview

Lastly, we have the situation where one account can impersonate another; will explain why this is important later.

This page describes how to allow members and resources to impersonate, or act as, an Identity and Access Management (IAM) service account. It also explains how to see which members are able to impersonate a given IAM service account.

— GCP — Managing Service Account Impersonation

Prerequisites

If you wish to follow along, you will need:

Two Google user accounts; in my case suzy.larkintuckerllc@gmail.com and fred.larkintuckerllc@gmail.com

A Google Cloud Platform (GCP) project owned by one of the accounts, in my case hello-accounts owned by suzy.larkintuckerllc@gmail.com

A Storage bucket in the GCP project, in my case hello-accounts-bucket

A service account in the GCP project, in my case hello-sa@hello-accounts.iam.gserviceaccount.com

The service account needs to have the permission, Project / Viewer; allows the service account to list the project’s buckets

A workstation with Python 3.x installed

The Python project, helloaccounts, downloaded and dependencies loaded by running

$ pip install -r requirements.txt

Please note: In order to create a clean virtual environment, one might want to use the pyenv tool.

User Account —Google Cloud SDK Command Line Tools

To authenticate as a user to the Google Cloud SDK Command Line Tools we execute:

$ gcloud auth login

...

You are now logged in as [suzy.larkintuckerllc@gmail.com].

Your current project is [None]. You can change this setting by running:

$ gcloud config set project PROJECT_ID

Please note: There is another command that will also authenticate a user in addition to setting other common configuration values; glcoud init.

We can observe this newly created configuration with:

$ gcloud config configurations list

NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION

default True suzy.larkintuckerllc@gmail.com

We can list the storage buckets in the project by executing:

$ gsutil ls -p hello-accounts

gs://hello-accounts-bucket/

Things to observe:

The user, suzy.larkintuckerllc@gmail.com, is the project’s owner and as such is authorized to list the project’s storage buckets

User Account— Google Cloud Client Libraries

Here we try to run our Python application and observe that we are not authenticated. Again, this is because authentication for Google Cloud Client Libraries is separate from Google Cloud SDK Command Line Tools.

$ python -m helloaccounts

...

raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)

google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

To authenticate as a user to the Google Cloud Client Libraries we execute:

$ gcloud auth application-default login --project hello-accounts

...

Credentials saved to file: [/home/sckmkny/.config/gcloud/application_default_credentials.json]

...

Quota project "hello-accounts" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

Things to observe:

Here we have to supply a project to attribute the API calls to; this is because Google limits (puts in quotas for) the number of API calls one makes

In the case of authenticating to Google Cloud SDK Command Line Tools, we did not need to supply a project. I assume that in this case the API calls are attributed to the project that the API calls are made to

Now that we are authenticated to Google Cloud Client Libraries, our Python application executes as expected.

$ python -m helloaccounts

hello-accounts-bucket

Service Account — Google Cloud SDK Command Line Tools

Using Google Cloud Console, we can create and download a key, a JSON file, for the service account. If we read the content of the JSON file, we will observe that the key’s expiration date is Dec 31, 9999. It is essentially a non-expiring key.

To authenticate as the service account to the Google Cloud SDK Command Line Tools we execute (changing out the account’s id and JSON file name as appropriate):

$ gcloud auth activate-service-account hello-sa@hello-accounts.iam.gserviceaccount.com --key-file=hello-accounts-54ae4707bd76.json

We can observe this overwrote our previous configuration with:

$ gcloud config configurations list

NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION

default True hello-sa@hello-accounts.iam.gserviceaccount.com

Please note: Here we simply chose to use a single configuration, default; we, however, could use multiple configurations if we wanted to.

We can list the storage buckets in the project by executing:

$ gsutil ls -p hello-accounts

gs://hello-accounts-bucket/

Service Account — Google Cloud Client Libraries

Because authentication for Google Cloud Client Libraries is separate from Google Cloud SDK Command Line Tools, we are still authenticated with the user account for Google Cloud Client Libraries.

To authenticate as a service account, we have to set an environment variable, GOOGLE_APPLICATION_CREDENTIALS, with the path to the downloaded JSON file.

$ export GOOGLE_APPLICATION_CREDENTIALS=/home/sckmkny/Desktop/hello-accounts-54ae4707bd76.json

Our Python application executes as expected.

$ python -m helloaccounts

hello-accounts-bucket

Please note: If we are not convinced that we are actually authenticating as the service account, we could temporarily remove the Project / Viewer permission from it and observe that the Python application will error; it takes about 30 seconds for permissions to propagate.

Service Account Impersonation — Google Cloud SDK Command Line Tools

As we saw earlier, the service account’s key, the JSON file, is essentially a non-expiring key which makes it a security risk.

Service accounts represent your service-level security. The security of the service is determined by the people who have IAM roles to manage and use the service accounts, and people who hold private external keys for those service accounts. Best practices to ensure security include the following:

- Use the IAM API to audit the service accounts, the keys, and the policies on those service accounts.

- If your service accounts don’t need external keys, delete them.

— GCP — Service Accounts

At the same time, we often want to test our applications on our workstations using the service account’s credentials. The solution, as we alluded to earlier, is to impersonate a service account.

To demonstrate this feature, let us first authenticate as our other user (one that currently has no permissions in our project) to the Google Cloud Client Libraries:

$ gcloud auth login

...

You are now logged in as [fred.larkintuckerllc@gmail.com].

Your current project is [None]. You can change this setting by running:

$ gcloud config set project PROJECT_ID

As before this overwrote our previous configuration:

$ gcloud config configurations list

NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION

default True fred.larkintuckerllc@gmail.com

We can verify that we currently do not have the proper permissions to list buckets in this project.

$ gsutil ls -p hello-accounts

AccessDeniedException: 403 fred.larkintuckerllc@gmail.com does not have storage.buckets.list access to the Google Cloud project.

To enable this user account to impersonate the service account, we add the Service Account Token Creator role for the user account on the service account.

Once those permissions propagate, which takes about one minute, we can then list the buckets in our project with the impersonation option.

$ gsutil -i hello-sa@hello-accounts.iam.gserviceaccount.com ls -p hello-accounts

WARNING: This command is using service account impersonation. All API calls will be executed as [hello-sa@hello-accounts.iam.gserviceaccount.com].

gs://hello-accounts-bucket/

We can also update our configuration to always use the impersonation:

$ gcloud config set auth/impersonate_service_account hello-sa@hello-accounts.iam.gserviceaccount.com

Updated property [auth/impersonate_service_account].

We list the buckets in the project without the impersonation flag and it still is successful.

$ gsutil ls -p hello-accounts

WARNING: This command is using service account impersonation. All API calls will be executed as [hello-sa@hello-accounts.iam.gserviceaccount.com].

gs://hello-accounts-bucket/

If we wish to update the configuration to not use the impersonation, we execute:

$ gcloud config unset auth/impersonate_service_account

Unset property [auth/impersonate_service_account].

Service Account Impersonation — Google Cloud Client Libraries

It is this last scenario that ultimately caused me to write this article as I could not find a simple solution on how to impersonate a service account when using Google Cloud Client Libraries. I thought this was odd as this would appear to be something one might commonly want to do.

I did, however, find a reasonably simple solution that involves a slight update to the application code.

To ensure that our code is working, we need to disable our previous mechanisms to authorize Google Cloud Client Libraries. Starting with removing the environment variable.

$ unset GOOGLE_APPLICATION_CREDENTIALS

And then revoking our user account authentication:

$ gcloud auth application-default revoke

You are about to revoke the credentials stored in:

[/home/sckmkny/.config/gcloud/application_default_credentials.json] Do you want to continue (Y/n)? Y Credentials revoked.

From the previous section, we know that the user that can impersonate the service account is still authenticated to Google Cloud SDK Command Line Tools.

The next step is to obtain a time-limited access token (which I believe expires in one hour) for the service account and store it into an environment variable, ACCESS_TOKEN.

$ export ACCESS_TOKEN=$(gcloud auth print-access-token --impersonate-service-account hello-sa@hello-accounts.iam.gserviceaccount.com)

WARNING: This command is using service account impersonation. All API calls will be executed as [hello-sa@hello-accounts.iam.gserviceaccount.com].

We then need to update our application code to use this environment variable; in particular the file helloaccounts/gc_sdk.py:

With this in place, we can successfully run our application as the service account.

$ python -m helloaccounts

hello-accounts-bucket

Wrap Up

For an article that I did not think I needed to write, it turned out to be rather lengthy. Hope you find it useful.