Running through the Google GCP Cloud Foundation Toolkit Setup
Tags: Google Cloud Platform, Infrastructure As Code, Software Development, Terraform, Infrastructure

The above graphic skipped the bootstrap. I know, mean right? But the bootstrap is a one-time step while the other three steps will be adapted continuously (albeit less frequently at the bottom)

0-Bootstrap

First, I forked the repo. However, we are using Github Actions and that means our service account key is added as a secret and exposed to terraform as an ENV variable. Since forks must be kept public, this couldn’t work. So, delete the fork and create a new repo instead. Otherwise, before you know it, someone will write a simple terraform snippet that extracts your sudo service account credentials and writes them to STDOUT, meaning it will end up in a public github comment. Bad Idea! So private repo it is.

Next, I decided to unify some variables. Since we’re using a monorepo, symbolic links have proven a good tool for us to keep terraform code DRY. A folder universal contains files that are, well, universal. For this project, that’s mainly universal.auto.tfvars and providers.tf . The variables below are all that’s needed to run through the 0-bootstrap step

org_id = "Y" billing_account = "X" group_org_admins = "gcp-organization-admins@dataminded.be" group_billing_admins = "gcp-billing-admins@dataminded.be" audit_data_users = "gcp-security-admins@dataminded.be" default_region = "europe-west3"

So, as always

terraform init

terraform plan

terraform apply

The first step however, is a bit special. After applying the code, a GCS bucket is created. Since the example repo comes with a commented out backend.tf , the README suggests to

Copy the backend by running cp backend.tf.example backend.tf and update backend.tf with your bucket from the apply step (The value from terraform output gcs_bucket_tfstate ) . Re-run terraform init agree to copy state to gcs when prompted

This is kinda neat. I have seen a lot of bootstrapping where people do a bunch of shell scripting to get started. This is cleaner.

This creates two projects (one for us, disabling the cloud build). More importantly, it creates a service account for which you can get an access key to use for the CI but also the manual applies for now.

gcloud iam service-accounts keys create account.json --iam-account org-terraform@gcp-foundation-seed-e3jd.iam.gserviceaccount.com

make sure you DO NOT commit this file to git. a quick

echo "**/account.json" >> .gitignore

should suffice.

1-Org

Alright, at this point I was motivated. It went smooth, next step!

cd ../1-org

cat README.md

Here I hit the aforementioned “each step is a separate repo” discrepancy. But that’s fine, I like our approach. With the linked files from universal, the directory looks as such

$ tree ./

├── backend.tf

├── folders.tf

├── log_sinks.tf

├── org_policy.tf

├── projects.tf

├── providers.tf -> ../_universal/providers.tf

├── README.md

├── terraform.tfvars

├── universal.auto.tfvars -> ../_universal/universal.auto.tfvars

├── variables.tf

└── versions.tf

Of course, you have to fix the backend.tf to point to the bucket created before. But the code is pretty explicity about that. Make sure to always read the README of each substep ;-)

Applying this code base however gave me trouble. We’re in Belgium, so we naturally went for europe-west1 which is the Belgium data center. Well, that causes issues!

Service availabilities for different EU regions

Turns out BigQuery is not supported in the Belgium region. Well, damn! Adjusting the default_region variable in the universal variables file means that the bootstrap step will be impacted! In fact, the state bucket will be impacted. But at this point, I have the 0-bootstrap and the 1-org state files in that bucket. So choosing the wrong region in the beginning is annoying! Make sure you choose a region that supports all the services to save yourself the trouble.

To fix this, I used the steps below

Now, we have the bucket in the new location and we can apply step 1-org

apply failed!

Damn, what now? Ah! Well, best to apply these changes with the aforementioned account.json account

export GOOGLE_APPLICATION_CREDENTIALS=<path_to_account_json>

This should work right?

Turns out, they are using what GCP calls service account impersonation . This makes sense on cloud build but we’re already running these steps as the service account. It can’t impersonate itself right?

So I removed all occurences of var.terraform_service_account and voilà!

2-Networks

Same steps as above, fix backend, link universal files, remove service account references. init, apply. Done

3-Projects

Last step, wuhu! Defining which projects you want in your org. This is also the place where you can import any existing projects. The sample repo utilizes the project-factory module but it is still a bit rough around the edges. In brief, they are using local exec too much to create projects, relying on a local python, gcloud and jq installation.

Anyways, I’d think up a nice folder structure for your projects here and then apply the code. Aftewards, you can add existing projects to the repo by using the terraform import functionality. New projects on the other hand can be created with the project-factory which will give you best-practices out of the box.

Conclusion

The cloud foundation toolkit by Google allows any experienced engineer to set up an enterprise cloud setup in less than a day. But as always, that’s more of a sales pitch than a statement to rely on. Do your homework and read the source code! Also, don’t just blindly apply the best practices but also know what they are. Still, it’s nice not having to reinvent the wheel. The old pattern was that people share best practices in blog posts and technical guidelines and then you had to translate that into your specific environment and apply them. Today, we’re taking a codified description of the best practices from the Google engineers themselves and then we apply that to our infrastructure! I’m so happy to cut out the middle man (me) and focus on the really important things instead. Like our weekly facts and breakfast sessions or creating real business value for our clients