Infrastructure as Code on Google Cloud Platform: Storage Buckets
By: David R Bayer
Tags: Infrastructure As Code, Google Cloud Platform, DevOps, Cloud Computing, Infrastructure

Part 2 in an n part series on what I’ve learned using Google’s deployment-manager

In my last post, I introduced Google’s deployment-manager and demonstrated setting up basic pub/sub topics and subscriptions. Now on to something more complex: storage buckets. After my experience with pub/sub I didn’t expect them to be that much more difficult, and in this case I was right.

My methodology so far has been to set up what I want by hand in Google Cloud Console, make sure it works, then try to recreate it with GDM (gcloud deployment-manager). To determine what settings I need in my config files, I have used a combination of the Google API documentation and gcloud * describe commands.

Storage bucket configuration can be a bit more complex than pub/sub. At a minimum you need the following YAML nodes:

name : In this case, the name of the resource block will also be the name of the resource. i.e. name: my-storage-bucket will create a bucket named my-storage-bucket

: In this case, the name of the resource block will also be the name of the resource. i.e. will create a bucket named my-storage-bucket type : The resource type, storage.v1.bucket in this case

: The resource type, in this case storageClass : Google recently updated their storage classes. As of now, you can specify MULTI-REGIONAL , REGIONAL , NEARLINE or COLDLINE . As of November 1, 2016, the default of STANDARD automatically translates to either MULTI-REGIONAL or REGIONAL depending on the location settings.

In addition to the above, you may also specify location , versioning , lifecycle , acl and other settings. Remember that some settings are incompatible. For instance, you may not specify location and storageClass: MULTI-REGIONAL . For a full list of the available options, the API documentation can be found here.

As a slight tangent, let’s talk about creating service accounts. It’s only a slight tangent. Really. Service accounts require very minor configuration:

- name: dav-test-serviceaccount

type: iam.v1.serviceAccount

properties:

accountId: dav-test-svc

displayName: dav-test-svc

A service account isn’t much good without a key, so then there’s this:

- name: dav-test-svc-key

type: iam.v1.serviceAccounts.key

properties:

parent: projects/my-project/serviceAccounts/dav-test-svc@my-project.iam.gserviceaccount.com

You might be saying to yourself, “Self, knowing that parent: value seems like a pretty rotten way to do things.” And you’d be right. You can specify values by reference for any resource that is included in the deployment file. So to put them together:

- name: dav-test-serviceaccount

type: iam.v1.serviceAccount

properties:

accountId: dav-test-svc

displayName: dav-test-svc

- name: dav-test-svc-key

type: iam.v1.serviceAccounts.key

properties:

parent: $(ref.dav-test-serviceaccount.name)

To tie this back in to the storage bucket topic, we can now grant permissions on our storage bucket to the new service account. One gotcha to remember when setting ACLs on storage buckets — when you set an ACL in deployment-manager, it replaces default or existing ACLs so specify all permissions you want to set.

The final config file that puts it all together:

resources:

- name: dav-test-serviceaccount

type: iam.v1.serviceAccount

properties:

accountId: dav-test-svc

displayName: dav-test-svc

- name: dav-test-svc-key

type: iam.v1.serviceAccounts.key

properties:

parent: $(ref.dav-test-serviceaccount.name) - name: dav-storagebucket-test-01

type: storage.v1.bucket

properties:

storageClass: MULTI_REGIONAL

versioning:

enabled: false

lifecycle:

rule:

- action:

type: Delete

condition:

age: 365

acl:

- role: OWNER

entity: "project-owners-123456789012"

- role: OWNER

entity: "project-editors-123456789012"

- role: READER

entity: "project-viewers-123456789012"

- role: WRITER

entity: "user-$(ref.dav-test-serviceaccount.email)"

# TODO: figure out how to get project-owners et.al. by reference

outputs:

- name: bucket

value: $(ref.dav-storagebucket-test-01.selfLink)

- name: svcacct

value: $(ref.dav-test-serviceaccount.email)

- name: svcacct-key

value: $(ref.dav-test-svc-key.privateKeyData)

# this will be base64 encoded

One thing I haven’t previously mentioned is the outputs block. This allows you to retrieve information about your deployment at the end of the run:

[dbayer@LVML201033: ~/tmp]$ gcloud deployment-manager deployments create --config=buckets.yaml dav-test-bucket

Waiting for create operation-1481485797164-543674aae03e1-60b947ad-bcac11a1...done.

Create operation operation-1481485797164-543674aae03e1-60b947ad-bcac11a1 completed successfully.

NAME TYPE STATE ERRORS INTENT

dav-storagebucket-test-01 storage.v1.bucket COMPLETED []

dav-test-serviceaccount iam.v1.serviceAccount COMPLETED []

dav-test-svc-key iam.v1.serviceAccounts.key COMPLETED []

OUTPUTS VALUE

bucket https://www.googleapis.com/storage/v1/b/dav-storagebucket-test-01

svcacct dav-test-svc@my-project.iam.gserviceaccount.com

svcacct-key ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsC...

# svcacct-key will be a long base64 encoded string [dbayer@LVML201033: ~/tmp]$

Unfortunately I haven’t figured out how to retrieve the private key yet for the service account, which limits the usefulness of creating the key. And I would also like to retrieve the project number and/or entities for ACLs by reference.

UPDATE: Marc Vieira-Cardinal showed how to do retrieve the private key in the comments. I have incorporated his information in the code block above. I also developed a method for retrieving the project number when using templates which I detailed here.

That about does it for storage buckets. Next up (assuming I don’t get distracted by something shiny): load balancers.