Data Ops – Git Actions & Terraform for Data Engineers & Scientists — GCP/AWS/Azure | by Gagandeep Singh | Towards Data Science

I strongly believe in POCing a new design or code template and get it review by other engineers because you never know what efficient nuances you are missing and it is always good to have a fresh set of eyes reviewing the code. And I tend to make POC as near to MVP (Minimum viable product) as possible to make myself and the team more confident of the design and to not waste any time in Regression testing later. It also helps in estimating my delivery task better and more accurately. But issues arise when I have to be dependent upon the DevOps team to ‘Infrastructure as code’ (IAC) my project in the dev environment. In the Prod environment, it is desirable to involve them in DevOps the infrastructure based on the best practices they have learned but in Dev, it can derail your MVP by just waiting for them to prioritize your task.
So a couple of years ago I started learning DevOps/DataOps and I started with Cloudformation (CFN) and Atlassian Bamboo since I was mostly working on AWS and the organization was using Bamboo. But lately, I got the chance to get my hands dirty in Terraform (TF) and Github Actions, because I was required to work on GCP, and dear oh dear it is way too easy to grasp and good to learn because with TF and Actions you can deploy in any cloud. And for a Data Engineer or Scientist or Analyst, it becomes really handy if you know an IAC tool. Since Github Actions sit closer to your code, it becomes all the more convenient for me.
So, I will break this down into 3 easy sections:
Integrating TF cloud to GithubGithub Actions workflow to run TF stepsOverview of TF files based on Best Practices
Integrating TF cloud to Github
Github Actions workflow to run TF steps
Overview of TF files based on Best Practices
Step 1: First thing first, TF has a community version so go ahead and create an account there. Here is the link: https://www.terraform.io/cloud. It will ask you to confirm your email address and everything so I am assuming that is done at the end of this step. Also, to complete the steps, it would be really great if you have your own account in the GCP or for that matter in any of the cloud. And your own Github account.
Step 2: TF has its hierarchy where each Organization has multiple workspaces and each Workspace has a one-to-one mapping with your IAC Github branch in repo where you will be pushing the updated Terraform files (Best practice). Files are written in Hashicorp Configuration Language (HCL) which is somewhat similar to YAML. So the idea is, whenever you will push a new update (or merge a PR) to a Github branch, Github-actions will execute and run the plan with a new changeset on TF cloud. Since TF cloud is linked to GCP (later), it will update the resource in GCP.
Create an organization and workspace inside of it in your Terraform community account.
Step 3: Assumingly, you have your own GCP account, create a service account in your project where you want the resource to be deployed, copy its key and put it in environment variable as GOOGLE_CREDENTIALS in Terraform variable. If you are using AWS then you need to put AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
This GCP Service account will be used to deploy the resource so it should have an Edit role or access level that has authority to deploy the resource.
You can also create a secret in Github secrets as shown below and pass the Credentials from there by using ${{ secrets.GOOGLE_CREDENTIALS }} in Github Actions as I did below.
Step 4: While creating a workspace, it will let you choose the github repo so that it can authenticate with it internally otherwise you have to generate a token in Terraform and save it in Github secrets to be used in ‘Setup Terraform’ step in Github actions.
The following Github actions script needs to be put in .github/workflow/ folder as anyname.yml. This has sequential steps in a particular job on what to do when someone pushes a new change in the repo. In the following, github actions will use bash in ubuntu-latest to checkout the code, setup terraform and run terraform init, plan and apply.
name: 'Terraform CI'on:  push:    branches:    - main  pull_request:jobs:  terraform:    name: 'Terraform'    runs-on: ubuntu-latest# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest    defaults:      run:        shell: bashsteps:    # Checkout the repository to the GitHub Actions runner    - name: Checkout      uses: actions/checkout@v2# Install the latest version of Terraform CLI and configure the Terraform CLI configuration file with a Terraform Cloud user API token    - name: Setup Terraform      uses: hashicorp/setup-terraform@v1# Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.    - name: Terraform Init      run: terraform init      env:        GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}# Checks that all Terraform configuration files adhere to a canonical format#    - name: Terraform Format#      run: terraform fmt -check# Generates an execution plan for Terraform    - name: Terraform Plan      run: terraform plan      env:        GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}# On push to main, build or change infrastructure according to Terraform configuration files # && github.event_name == 'push'      # Note: It is recommended to set up a required "strict" status check in your repository for "Terraform Cloud". See the documentation on "strict" required status checks for more information: https://help.github.com/en/github/administering-a-repository/types-of-required-status-checks    - name: Terraform Apply      if: github.ref == 'refs/heads/master'      run: terraform apply -auto-approve      env:        GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
Now terraform only reads those files that have .tf extension to it or .tfvars but there are some files that it needs to be named as is whereas others are just resource abstraction and will be executed in the no order unless you mention the dependencies using depends_on. Main.tf, variables.tf and terraform.tfvars are the files that need to be created with the same name.
main.tf: All the resources can be mentioned here with its appropriate configuration values. So for example if you want to create a storage bucket then it can be written in HCL as below:
resource “google_storage_bucket” “bucket” {     name = “dev-bucket-random-12345”}
variables.tf: As the name suggests, it has variables like below with its default value which you want to re-use in other resources or .tf file like ${var.region}. This is more like a convention to put the variables in different variables.tf file but you can put it in main.tf too.
variable “region” {     type = string     default = “australia-southeast1”}
terraform.tfvars: This will have the actual values of the variables defined above. Essentially, if region is not mentioned below then it will take the above default value.
project_id = “bstate-xxxx-yyyy”region = “australia-southeast1”
The rest of the files are to abstract different types of resources into different files. For example networks.tf will have VPC and Subnet resources, stackdriver.tf will have alerts and monitoring dashboards, dataproc.tf will have cluster and nodes resources in it and similarly for firewalls, GKE, etc.
For all those who don’t know, TF and CFN have documentation where these predefined functions or Resource configurations are there to help us understand what options mean. Following is of GCP: https://registry.terraform.io/providers/hashicorp/google/latest/docs
In the following example, a monitoring channel is created for the Stackdriver alerts. All the fields are self-explanatory and come from the GCP Terraform documentation and with the help of the output variable ‘gsingh_id’, you can directly use it in any .tf file or if you don’t want to specify the output, you can directly use it like this: google_monitoring_notification_channel.gsingh
resource "google_monitoring_notification_channel" "gsingh" {  display_name = "xxxx@yyyy.com"  type = "email"  labels = {    email_address = "xxxx@yyyy.com"  }}output "gsingh_id" {  value = "${google_monitoring_notification_channel.gsingh.name}"}
Below a Subnet is getting created and VPC is specified as a dependency for the subnet in a similar fashion.
resource "google_compute_subnetwork" "vpc_subnetwork" {  name          = "subnet-01"  network       = google_compute_network.vpc_network.self_link  private_ip_google_access = true  ip_cidr_range = "10.xx.xx.xx/19"  region        = "australia-southeast1"  depends_on = [    google_compute_network.vpc_network,  ]}
Now as explained earlier, whenever you push a new change to this repo, Git actions will checkout the code from the master branch and run Terraform Init, Plan and Apply to deploy the changes in the cloud. In the next couple of days, I will be publishing a series of articles on how to Deploy a Flink application on GKE cluster through Git actions which will also let you know how to build a Scala app using Git Actions. So stay tuned.