fabiochiu/medium-articles huggingface dataset:
How to Automate Governance Best Practices With Google Data Catalog and Terraform
By: Marcelo Costa
Tags: Data Governance, Data Catalog, Google Cloud Platform, Terraform, Data Management

Disclaimer: All opinions expressed are my own, and represent no one but myself…

There’s extensive documentation on what IAM Roles are available for Google Data Catalog. But when you are getting started with your data governance journey, you probably have wondered what kind of access controls are needed and who should be granted them in your organization…

What end user should be able to discover my data assets?

Who should be able to classify and add tags to them?

And finally, be able to create templates and set standards for the data classification process?

This can get really complex, so in this blog post, we will start by looking at the access controls on top of metadata, which is Google Data Catalog playing field.

And what if I told you that we can automate all that by using Terraform? Sounds good, right? So let’s get started!

The solution

Data Governance Architecture

This is simply a suggestion on how to work with Data Catalog. To start off, let’s say you have some common templates that will be used to create tags in different projects.

For that we need two different pieces:

The Tag Central Project

This is where we store all the common resources, like Tag Templates, Policy Tags, and Custom entries. So we don’t duplicate those, are charged only once, and have a much easier time when managing and making changes to them.

To showcase this, in the Terraform sample we will create 4 Tag Templates in the Tag Central project:

★ Data Engineering Template

★ Derived Data Template

★ Data Governance Template

★ Data Quality Template

A Group of Analytics Projects

By Analytics Projects, I mean those that will store data assets, such as Big Query tables, Pub Sub topics, or any other resource managed by Data Catalog.
Deploying Helm Charts into K8S with Terraform
By: Nicholas Lu
Tags: Kubernetes, Helm, Eks, DevOps, Terraform

Photo by Andrew Neel on Unsplash

“Look mom, no hands on Console”, Deploying Helm charts into Kubernetes has never been easier with Terraform

I have just started my DevOps journey with Terraform not long ago and there are too much deployment methods into Kubernetes cluster. The existence of multiple deployment guide and pattern makes it easy for developer to get their code into the cluster with the requirement of knowing the ins and out of Kubernetes.

There are plenty of mode of deployment such as:

1) Creating a surrogate container to run CI/CD tasks via platform like Gitlab/Github Actions/Jenkins etc

2) Direct injection of manifest file/deployment on CI tools

3) Helm, a tool that distills deployment component together into a single package(includes deployment, ingress, service, configmap etc)

In this post, I am going to show how to deploy any charts into your cluster via terraform.
This is a very interesting discussion , but I wonder if something like Terraform is a better…
Tags: Iaac, Terraform

re and more out of …t have the source. Perhaps it was done by somebody else, who preferred a completely different tool. I’ve often seen a project’s documentation get more and more out of date because nobody can update the damn diagrams. If it’s just source code, your chances get a lot better.
terraform ignore_changes lifecycle to avoid re-creation
By: Ismail Yenigül
Tags: Lifecycle, Terraform

If you create an aws_directory_service_directory and change password parameter, terraform will destroy and create AD resource.

provider "aws" {

region = "us-west-2"

} resource "random_password" "mypassword" {

length = 20

special = true

override_special = "_$%"

min_special = 1 } resource "aws_directory_service_directory" "mysimplead" {

name = "test.simplead"

type = "SimpleAD"

description = "my simple AD"

password = random_password.mypassword.result

alias = "mysimple"

size = "Small" vpc_settings {

vpc_id = "vpc-005d9xyz2658a45"

subnet_ids = ["subnet-015fcx166c80407", "subnet-0d0f1x6d5919b8"]

} }

I just edited mypassword values then run terraform apply

# aws_directory_service_directory.mysimplead must be replaced

-/+ resource "aws_directory_service_directory" "mysimplead" {

~ access_url = "mysimple.awsapps.com" -> (known after apply)

alias = "mysimple" # random_password.mypassword must be replaced

-/+ resource "random_password" "mypassword" {

~ id = "none" -> (known after apply)

~ length = 20 -> 15 # forces replacement

lower = true

min_lower = 0

min_numeric = 0

min_special = 1

min_upper = 0

number = true

override_special = "_$%"

~ result = (sensitive value)

special = true

upper = true

} Plan: 2 to add, 0 to change, 2 to destroy. Do you want to perform these actions?Terraform will perform the actions described above.

Only 'yes' will be accepted to approve. Enter a value: yes aws_directory_service_directory.mysimplead: Destroying... [id=d-92670bef7d]

aws_directory_service_directory.mysimplead: Still destroying... [id=d-92670bef7d, 10s elapsed]aws_directory_service_directory.mysimplead: Still destroying... [id=d-92670bef7d, 20s elapsed]aws_directory_service_directory.mysimplead: Still destroying... [id=d-92670bef7d, 30s elapsed]

To avoid deletion on password change, you can add the following policy to resource “aws_directory_service_directory”

lifecycle {

ignore_changes = [password]

}

Ismail YENIGUL

Devops Engineer
Terraform, AWS Batch, and AWS EFS
By: Joseph Min
Tags: AWS, Aws Efs, Terraform, Aws Batch

I recently came across an issue where I had to stand up a scalable compute pipeline and make it easy to deploy for others to stand up themselves.

For a myriad of reasons (I won’t give you the story of the long road filled with tech hardships or any of that), I went with Terraform for my declarative infrastructure, AWS Batch for my compute, and AWS EFS as my storage. However, it was pretty annoying to get everything together because there’s not a ton of documentation on how to make it all work. So… eggs.

You know how I’m filled with rage? I have no outlet for it so… eggs.

I’m going to describe how everything works here, but if you just want the code snippets (I know that’s all I ever care about) skip down to the Terraform section.

Basically, the Terraform scripts below (which I’m going to assume you know how to run, but if not, check out their docs) will stand up the AWS resources for you to have an Elastic Filesystem (EFS) mounted in your AWS Batch job so that you can write common resources to that filesystem (maybe a large dataset that you want to read from often) and have them persist and be accessible from all future jobs. This lets you not have to pull this large dataset from S3 with every job, and instead instantaneously mount the filesystem to read directly from that dataset.

Let’s imagine a scenario: You have a large dataset. Like, HUGE. You pull in or download that large dataset the first time your job runs and write it to your mounted EFS at /mnt/efs/big.data . Then as part of that same job, you search through big.data for some matching patterns or push them through some pipeline with parameters as set in your job at runtime. You complete the computation part, write some output, which you then push to an S3 bucket for results.

Now, you want to run the same pipeline but with slightly different parameters. Actually, you want to run it 100 more times all at the same time, each with slightly different parameters but still all reading from the same dataset. Instead of downloading this large dataset either from the internet or pulling it in from S3 every time, your jobs will automatically mount the filesystem in which big.data exists. EFS handles file-locking and concurrent reads so you don’t have to worry about data access management, and your jobs don’t have the hour-long overhead of downloading the dataset every time.

The idea is cool, but the documentation is sparse, so… Let’s get down to it!
Using Amazon EBS gp3 to Reduce your Storage Spend
By: Thomas Gallagher
Tags: Ebs Volume, DevOps, AWS, Cloud, Terraform

Re:Invent

Today, at AWS Re:Invent, Amazon Web Services revealed that they have been working on “high performance block storage,” with more iOPS, aptly named gp3. During the keynote address, AWS CEO Andy Jassy announced that AWS users would be able to “…provision Performance apart from Capacity.”

The new gp3 volumes are already available in the AWS console, and if you’re using Terraform as I am, you can easily convert your non root_block_device volumes to gp3 by adding the type variable to your code, as evidenced below.

availability_zone = module.vpc.availability_zone[0] size = “30” device_name = “xvdb” instance_id = module.aws_test_instance.id[0] type = “gp3”

Adding the “type” variable will produce the below once you run a Terraform Plan:

Pricing and iOPS

Perhaps the most impactful piece that I have not mentioned yet: the gp3 volume type offers AWS users a savings of 20% per gigabyte over the gp2 volume type, which almost makes it a no brainer to start switching your instances over to gp3.

With 3000 free iOPS, gp3 is superior to gp2 in almost every capacity, offering users a 1000x increase in base throughput capacity.

Conclusion

You can find more information on the release of gp3 in AWS’s own blog post about it today, which can be found here.

It is clear to this avid AWS consumer that AWS is determined to continue their tireless battle to claim more and more cloud market share, and I, for one, am extremely excited at all of the new offerings that AWS unveiled during AWS Re:Invent today.

If you enjoyed reading this article, please feel free to check out my other Medium articles, and to visit my LinkedIn, where I am active and always willing to chat about all things AWS.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬
Separating development and production environment in terraform
By: Prabesh Thapa
Tags: Better Programming, Tfstate, Startup, Terraform, Infrastructure As Code

Today, we are going to work on how we can separate development and production environments.

Terraform recently introduced a feature called Workspaces. This feature allows you to separate your dev environment from other environments like corp and prod.

By default terraform creates a workspace named “default”. In order to create a new workspace, after you run “terraform init” run

terraform workspace list

This command will show you the list of Terraform workspace. To create a new workspace,

terraform workspace new <workspace name>

This will create a local directory “terraform.tfstate.d” directory where the state file for your workspace would be stored.

To select a workspace that you created, run the following command

terraform workspace select <workspace name>

What if there are multiple environments and multiple people working on terraform?

Here terraform backend and remote states comes to the rescue.

You can see how to use remote state from this article

Once you followed this article and created an S3 bucket, create a folder named “env:” and now run terraform init and create workspace using the following command

terraform workspace list terraform workspace new <workspace name> terraform workspace select <workspace name>

For this article, I will create 3 environments (dev, corp, prod)

so first I will create a backend with a remote state

<script src=”https://gist.github.com/pgaijin66/dbe5341046f41ac04a864cf22b3428ed.js"></script>

Then create workspaces

terraform init terraform workspace new dev terraform workspace new corp terraform workspace new prod

Now, if you check in the s3 bucket env: folder, you can see the workspaces and state files stored.
Terraform and its basic commands
By: Pratyush Pandey
Tags: Tool, AWS, Commands, Terraform

Introduction to Terraform

Terraform is defined by HashiCorp, the creator of Terraform, as a tool for building, changing, and versioning infrastructure safely and efficiently. That sounds like an infrastructure as code solution, because Terraform is an infrastructure as code solution.Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can help with multi-cloud by having one workflow for all clouds. The infrastructure Terraform manages can be hosted on public clouds like Amazon Web Services, Microsoft Azure, and Google Cloud Platform, or on-prem in private clouds such as VMWare vSphere, OpenStack, or CloudStack.

Features of terraform

The key features of Terraform are:

Infrastructure as code

Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.

Execution plans

Terraform has a “planning” step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure.

Change Automation

Complex change sets can be applied to your infrastructure with minimal human interaction. With the previously mentioned execution plan and resource graph, you know exactly what Terraform will change and in what order, avoiding many possible human errors.

Five Commands of terraform are as follows:

1. Terraform show: The terraform show command is used to provide human-readable output from a state or plan file. This can be used to inspect a plan to ensure that the planned operations are expected, or to inspect the current state as Terraform sees it.

Command: terraform show -json This will show the output in JSON representation of the state.

2. Terraform get: The terraform get command is used to download and update modules mentioned in the root module.

Command: terraform get [options] [dir]

3. Terraform providers: The terraform providers command prints information about the providers used in the current configuration. This command gives an overview of all of the current dependencies, as an aid to understanding why a particular provider is needed.

Command: terraform providers [config-path]

4 .Terraform plan: The terraform plan command is used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.

Command: terraform plan [options] [dir]

5. Terraform apply: The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan.

Command: terraform apply [options] [dir]
System Provisioning Management Tool
By: Nirmaljeet Singh
Tags: DevOps, Terraform, Infrastructure As Code

System Provisioning Management Tool

Terraform is an open source tool for infrastructure provisioning created by HashiCorp. It provides Infrastructure as code allowing you to automate and manage your cloud, infrastructure, platform and your services that run on the platform. Terraform can manage existing and popular service providers(aws, azure, GCP etc.)

You do not have to prepare infrastructure like private network space, ec2 server instances, installing docker and other tools and security. Terraform does all that for you by preparing the whole infrastructure using terraform scripts. Thus, it is a software tool that provides Infrastructure as code.

Terraform is declarative which means you define what you want.

Below mentioned is an example where you define the instances that you want to create on aws cloud.
Using terraform module from a specific branch
Tags: AWS, Linux, Terraform Modules, Terraform

This is mostly for myself as notes for work, but hope you can find it useful.

Sometimes if you are working on a terraform module and committing new code for the module to a specific branch, the need will arise to work off that specific branch when downloading and using the module.

I couldn’t really find much info on this on the google, so thought I’d post it here for myself for future reference:

module "testing_datadog" { source = "git@github.com:somerepo/tf-module-datadog?ref=FEATURE_SWITCH_BETWEEN_SCALE_TYPE"

If you are just referencing the “master” branch which github is trying to replace with the “main” branch you can do:

module "testing_datadog" { source = "git@github.com:somerepo/tf-module-datadog"

Remember to do a :

terraform init

everytime that you make changes to that branch and you would like to pull in those changes to your module.
System Provisioning Management Tool(Terraform) And Its 5 Popular Commands
By: Parth Rai
Tags: Terraform

System Provisioning Management Tool(Terraform) And Its 5 Popular Commands

Terraform is an open source Infrastructure as Code tool which is created by HashiCorp. It is currently the most popular automation tools available.

Use Infrastructure as Code to provision and manage any cloud, infrastructure, or service. It is a tool for building, changing, and versioning infrastructure safely and efficiently. It can manage existing and popular service providers as well as custom in-house solutions. Configuration files describe to Terraform the components needed to run a single application or your entire datacenter.

Terraform uses a Simple syntax, it can provision infrastructure across multiple cloud on premises data centers and can efficiently re-provision infrastructure in response to configuration changes.

Data source to access information about an existing Image.

TOP 5 POPULAR COMMANDS OF TERRAFORM

DESTROY- This command is used to destroy the Terraform-managed infrastructure. This command accepts all the arguments and flags that the apply command accepts, with the exception of a plan file argument. This command is the reverse of terraform apply in that it terminates all the resources specified by the configuration. It does not destroy resources running elsewhere that are not described in the current configuration.

2. Graph- This command is used to generate a visual representation of either a configuration or execution plan. The output is in the DOT format, which can be used by graph viz to generate charts.

3.Terraform init: The terraform init command is used to initialize the working directory in which the terraform configuration files are present these files includes such as main.tf, variables.tf etc. This is the first command that should be run after writing a new terraform configuration or cloning an existing one from version control.

4.Terraform plan: The terraform plan command is used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.

Command: terraform plan [options] [dir]

5.Terraform apply: The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan.

Command: terraform apply [options] [dir-or-plan]

References:
Terraform remote backend for cloud and local with Azure DevOps Terraform Task
By: Pr Code - Christopher Pateman
Tags: Azure Devops, Azure Devops Pipeline, Azure, Terraform Provider, Terraform

When working with Terraform, you will do a lot of work/testing locally. Therefore, you do not want to store your state file in a remote storage, and instead just store it locally. However, when deploy you don’t want to then be converting the configuration at that point and can get messy working with Azure DevOps. This is a solution that works for both local development and production deployment with the Azure DevOps Terraform Task.

The official Terraform Task in Azure DevOps by Microsoft is https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks

When using this task you configure the cloud provider you will be using as a Backend service like Azure, Amazon Web Services (AWS) or Google Cloud Platform (GCP). These details can be used to configure the Backend Service to store the State file, but they require the Terraform code to implement the service.

You can see all the different types here: https://www.terraform.io/docs/backends/types/index.html

For this walk through I will use the Azure Resource Manager, which uses an Azure Storage Account, as the example, but as mentioned this can be used in any provider.

https://www.terraform.io/docs/backends/types/azurerm.html

This would be the standard Terraform configuration you would need for setting up the Backend Service for Azure:

terraform { backend "azurerm" { resource_group_name = "StorageAccount-ResourceGroup" storage_account_name = "abcd1234" container_name = "tfstate" key = "prod.terraform.tfstate" } }

When using this locally, you don’t want any of this in your main.tf Terraform file else it will error with no detail or add the state to the Azure Storage Account. Therefore, locally you will not add this.

Instead during the deployment using the Azure DevOps Pipelines, we will inject the configuration. This will be done, by inserting a backend.tf file using PowerShell. Within the file, we will inject the configuration, but we don’t need all the parameters as they are inserted by the Terraform task.

We will inject just:

terraform { backend "azurerm" { } }

Which as a single like string we will need to stringify it to:

"terraform { `r`n backend ""azurerm"" {`r`n} `r`n }"

Using the PowerShell task we can then check for if the file already exist and if not then inject it into the same location as the main.tf file. This then causes when Terraform runs to process it with a Backend Service and with the Azure details we have provided in the Task.

View the full formatted result in the original post > http://prcode.co.uk/2020/11/25/terraform-remote-backend-with-azure-devops-terraform-task/
Collaborative IaC using Terraform
By: Prabesh Thapa
Tags: Iac, S3, Dynamodb, Startup, Terraform

Terraform is taking over the market and since it is cloud-agnostic, you can use the same language paradigm to provision infrastructure in any cloud environment.

This is really good when you are provisioning in your local machine, but what happens when you want to let multiple developers work on the same code? One way is sharing your state file but this is not recommended as terraform state holds some critical information like passwords, IPs.

So how do we solve this?

This can be done using a terraform remote state.

Remote state:

Using a remote state you can store your state configuration file on the cloud and the same state files can be used by different developers to provision infrastructure.

In order to define the terraform remote state your need to create a file named “backend.tf”, this will hold all the backend information and configuration of your terraform.

# To hold state file locks

terraform { # Use latest terraform version and always pin versions required_version = "=0.13.4" backend "s3" { # To store state files bucket = "terraform-live" key = "terraform.tfstate" region = "us-west-2" # To hold state file locks dynamodb_table = "terraform-live-locks" encrypt = true } }

Once you have it ready, you need yo created a s3 bucket named “terraform-live" and DynamoDB table named “terraform-live-locks" with key “LockID” which would hold the locks when you run terraform operations.

After this, run terraform init and you are ready to go.
Introduction to terraform
By: Rishabh Singh
Tags: AWS, Terraform

Terraform

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can help with multi-cloud by having one workflow for all clouds. The infrastructure Terraform manages can be hosted on public clouds like Amazon Web Services, Microsoft Azure, and Google Cloud Platform, or on-prem in private clouds such as VMWare vSphere, OpenStack, or CloudStack.

Terraform is defined by HashiCorp, the creator of Terraform, as a tool for building, changing, and versioning infrastructure safely and efficiently. That sounds like an infrastructure as code solution, because Terraform is an infrastructure as code solution.

Features

The key features of Terraform are:

Infrastructure as code

Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.

Execution plans

Terraform has a “planning” step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure.

Change Automation

Complex change sets can be applied to your infrastructure with minimal human interaction. With the previously mentioned execution plan and resource graph, you know exactly what Terraform will change and in what order, avoiding many possible human errors.

Five Commands of terraform which are used commonly are as follows:

Terraform plan: The terraform plan command is used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.

Command: terraform plan [options] [dir]

2. Terraform apply: The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan.

Command: terraform apply [options] [dir]

3. Terraform show: The terraform show command is used to provide human-readable output from a state or plan file. This can be used to inspect a plan to ensure that the planned operations are expected, or to inspect the current state as Terraform sees it.

Command: terraform show -json This will show the output in JSON representation of the state.

4. Terraform get: The terraform get command is used to download and update modules mentioned in the root module.

Command: terraform get [options] [dir]

5. The terraform providers command prints information about the providers used in the current configuration. This command gives an overview of all of the current dependencies, as an aid to understanding why a particular provider is needed.

Command: terraform providers [config-path]
Terraform- A requisite for DevOps Spectrum
Tags: Iac, Automation Tools, DevOps, Terraform

DevOps chain has a lot of tools, which make software development lifecycle quite easy. Terraform is one of the most important tools of the provisions sector in DevOps today. But why is it used?

Problems before DevOps in general:

Before DevOps, the process of software development was so slow and sometimes became a very painful process. The different mindsets, targets, and responsibilities of Developers and Ops people created a huge gap between them.

This problem was seldom solved with the introduction of DevOps and its tools, which managed Devs and Ops with tools like version control, CI/CD, automated testing and deployment provisioning. This whole process made software development less tedious and agile. Here comes the role of Tool named terraform which can easily manage all the automation needed for provisioning.

Terraform:

It is a tool for developing, modifying and versioning the infrastructure safely with efficiency. Terraform is the first ever tool that supports multi cloud providers with ease. It was developed by Harshicop and written in Go. Terraform has become more popular due to its simple syntax which supports modularity with multi cloud.

Why Terraform?

As its well knows that the goal of DevOps is to perform efficient software delivery therefore we need some tools to make it happen quickly. This is where Terraform comes which help companies with automation and infrastructure as code(iac). Your whole cloud’s infrastructure can be managed by single tool including instances, volumes, storage, servers, monitoring, APIs and what not, with just few commands written in the script.

Let us see few reasons for justifying “why Terraform”:

Terraform lets you define the infrastructure in code, and hence will enable you to track the changes with utmost ease. It is the only tool which supports multiple cloud providers. While other alternatives only support single cloud provider. It has massive community support, which enables a lively discussion on any problem encountered while working.

Now lets talk about some popular Terraform commands for aws cloud provider:

terraform init:

this is the command which initialises to download respective provider plugin for cloud in your local system.

2. terraform plan:

its the command to see what you are gonna deploy.

3.terraform apply:

the plan command tells what terraform is gonna apply. if all ok, then this command will apply the changes.

4. terraform validate: this command validates the configuration files. it checks whether the configuration files in the directory are syntactically valid or not. It is safe for running it automatically.

5. terraform destroy:

with this command, the infrastructure created by terraform will be vanished. so use this command very carefully.

Lastly, terraform makes a great tool when it comes to DevOps hence getting a lot of popularity.
System Provisioning Management Tool
Tags: Terraform

Terraform

If we get into the definition of provisioning then, provisioning is a process of setting up IT infrastructure. It includes steps that are necessary to manage the access to data and resources and also make them available to the user. Provisioning and Configuration are two different steps, both are part of deployment. Provisioning is a step before configuration.

Terraform is a Infrastructure as a code tool which is open source and is created by HashiCorp. Terraform is more of a declarative coding tool, it allows the developers to use HashiCorp Configuration Language which is a high level configuration language. This language is used to describe the desired end state. Since Terraform is a IaC tool, therefore the infrastructure can be coded in a way to make it fast, automated and repeatable.

Why Terraform?

Open source- It is very easy to find plugins, extensions and professional support because Terraform is backed by a large community of contributors. This also means terraform evolves very speedily.

Platform Agnostic- It can be used with any cloud service provider.

Immutable infrastructure- The problem with the mutable infrastructure is the configuration drift. In terraform with each change to the environment, the current configuration is replaced with a new one that accommodates the changes.

Modules- Package and Reuse code

Another benefit that Terraform provides is that of packaging and reuse the common code in forms of module.. The use of modules is to provide a simple interface for creating resources, this is done by producing inputs and returning outputs. The benefit of this is that it helps in cutting down of duplicate code.

Terraform vs Ansible

Both Terraform and Ansible are Infrastructure as a Code tools, there are some differences between the two-

1. Terraform is a descriptive tool, while Ansible is both declarative and procedural configuration tool. Procedural configuration is more work but it also provides more control.

2. Terraform is open source while Ansible is a product of RedHat.

5 main Terraform commands are-

Now run the following commands

terraform init

It initializes working directory containing terraform configuration files(.tf files) and it is safe to run this command multiple times.

2. terraform validate

It checks if terraform scripts have no syntax errors and are internally consistent.

3. terraform plan

It create execution plan that helps you check whether execution plan matches your expectations.

4. terraform apply

It applies changes to reach the desired state of the configuration.

5. terraform destroy

It terminates resources defined in your Terraform configuration.
Kubernetes cluster on Azure, AWS, GCP with Terraform and Seldon
By: Xin Cheng
Tags: Ml Model Deployment, Cloud, Kubernetes, Terraform

Major cloud providers provide managed Kubernetes cluster offering and cli tool to create it easily. However, lots of companies are using are using IaC tool terraform for automating infrastructure provisioning. In this article, let’s use terraform to create Kubernetes cluster on different clouds, and quickly test a machine learning model deployment using Seldon

Install Terraform CLI

On Ubuntu, it can be as simple as

sudo snap install terraform

Install kubectl

Install cloud CLI (command line interface) and authenticate

Azure

GCP

AWS

Create Kubernetes cluster

Overall steps

Define VPC/virtual network/firewall rules Define kubernetes cluster (master node/control plane, cluster name, location, VPC/subnet) Define node pool (worker node/data plane, specification, node count)

AKS

difference: service principal

GKE

EKS

Model deployment

Install s2i

Install seldon-core

sklearn_iris model deployment

Run following command to train model

python3 train_iris.py

Now run

s2i build -E environment . seldonio/seldon-core-s2i-python3:0.18 seldonio/sklearn-iris:0.1

Local docker test



curl -s docker run --name "iris_predictor" -d -p 5000:5000 seldonio/sklearn-iris:0.1curl -s http://localh ost:5000/predict -H "Content-Type: application/json" -d '{"data":{"ndarray":[[5.964,4.006,2.081,1.031]]}}'

Setup seldon-core operator and ambassador





kubectl apply -f

kubectl apply -f # helm issue https://github.com/datawire/ambassador-chart/issues/90 kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-crds.yaml kubectl apply -f https://www.getambassador.io/yaml/ambassador/ambassador-rbac.yaml kubectl config set-context $(kubectl config current-context) --namespace=default helm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set ambassador.enabled=true --set usageMetrics.enabled=true --namespace seldon

Deploy model

kubectl create -f sklearn_iris_deployment.yaml

kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-deployment-example \

-o jsonpath='{.items[0].metadata.name}')

Test model



kubectl port-forward svc/sklearn-iris-example-sklearn-iris-example 8000:8000

curl # through seldon deployment podkubectl port-forward svc/sklearn-iris-example-sklearn-iris-example 8000:8000curl http://localhost:8000/api/v1.0/predictions -H "Content-Type: application/json" -d '{"data":{"ndarray":[[5.964,4.006,2.081,1.031]]}}'

kubectl port-forward $(kubectl get pods -l # through ambassadorkubectl port-forward $(kubectl get pods -l app.kubernetes.io/managed-by=getambassador.io -o jsonpath='{.items[0].metadata.name}') 8003:8080 curl -s http://localhost:8003/seldon/default/seldon-deployment-example/api/v1.0/predictions -H "Content-Type: application/json" -d '{"data":{"ndarray":[[5.964,4.006,2.081,1.031]]}}'

Result

{"data":{"names":["t:0","t:1","t:2"],"ndarray":[[0.9548873249364059,0.04505474761562512,5.

7927447968953825e-05]]},"meta":{}}

Access ambassador diagnostic UI
Is Scalr the Best Alternative to Terraform™ Enterprise?
By: J.B. Beck
Tags: AWS, DevOps, Terraform, Cloud

tl;dr Scalr is the best alternative to Terraform Enterprise if you want transparent pricing, Open Policy Agent integration or a template registry.

So you’ve evaluated Terraform Enterprise and dropped it because the cost was prohibitive? Scalr is an alternative that has feature parity with Terraform Enterprise and full CLI support. It also has some key differentiators and quality of life features that make it even better.

Key Differences Between Scalr And Terraform Enterprise

Public Pricing

For an on-premise installation of Scalr, the pricing is per workspace and a workspace will cost a maximum of $40/month. Scalr has a $25,000/year minimum spend and offers volume discounts:

0–100 workspaces: $40/workspace/month

101–500 workspaces: $32/workspace/month

501–1000 workspaces: $25/workspace/month

1001+ workspaces: $18/workspace/month

Scalr offers payment terms discounts. If you commit for 3 years, you get a 10% discount.

Example: 150 workspaces in a 3 year contract will cost (100*40+50*32)*0.9 = $5,040/month

Open Policy Agent Integration

Scalr uses OPA for policy as code instead of Sentinel. OPA is domain agnostic while Sentinel only applies to Hashicorp products. No need to have a different policy language for every service you use. OPA is the tool of choice for a unified policy framework across the cloud native stack.

Template Registry

Scalr lets users deploy preconfigured Terraform templates themselves. Users don’t need prior Terraform knowledge or going through a ticketing system.

Hierarchical Management Model

Scalr allows the sharing of objects across organizations. An object can be a VCS configuration, a policy, a variable … No need to go into each organization to create a module or a VCS provider. Scalr doesn’t charge per user per organization but per user across all organizations.

Policy Update Blast Radius

It is possible to visualize the blast radius of policy updates to your existing and future deployments with policy previews. Create a pull request against your policy code and understand the impacts in the Scalr UI. For example, admins get a list of workspaces impacted based on the policy change. Engineers have the ability to preview the impact of the new policy when deploying infrastructure.

Getting started with Scalr

Sign up for a free account on our SaaS version to make up your mind and see if Scalr meets your needs. You don’t need a credit card for that, and you can invite up to 4 team members. If you like what you see and want to move forward, reach out to us and we’ll be happy to demo Scalr to you and assist you with a POC.

Terraform™ is a registered trademark of Hashicorp
Cleaning AWS S3 multipart leftovers
By: Adrian Macal
Tags: S3, AWS, Terraform

Many tools and frameworks perform uploading files to S3 in parts to gain performance or overcome 5GB limitation. If the process is terminated and not completed it may still occupy storage space.

CLI

You can list all multipart uploads in bash for each bucket:

aws s3api list-multipart-uploads --bucket=

You can even remove them manually by calling:

aws s3api abort-multipart-upload --bucket= --key= --upload-id=

Automation

AWS already provides solution for it which may be just enabled. I was always told that S3 Lifecycle Rules are for moving objects between storage classes. It turns out you can do much more with it. Let’s enable incomplete multipart upload deletion.

Outcome

If everything went right you will see in the Console newly added and enabled rule.
AWS Global Accelerator — Http to Https Redirection(terraform)
By: Subhasis Ray
Tags: AWS, Global Accelerator, Http To Https Redirect, Terraform

We had requirement to make our Application Loadbalancer highly available and also to remove dependency from the DNS provider. Let me just explain little further. We have our DNS provider which is not AWS route53, so when we drop into any incident or DR(Disaster Recovery) then we have to reconfigure DNS with new ALB. Because current ALB will not be functioning. This will take some time to reflect as well. Or else we have to create DR ALB earlier and keep it ready and configure with DNS provider with traffic dial 0%. Some of the DNS provider does not support this feature.

So we come up with Global accelerator, which also used AWS global network to route the traffic to optimal regional endpoint based on different factors(client location, policies). While doing so we found that our Global accelerator not transferring traffic from http to https, as we created the Global accelerator with terraform. Though we have redirection in ALB level.

Below changes has been done in Global accelerator to transfer traffic. Find the modules below for terraform.

resource “aws_globalaccelerator_accelerator” “global_accelerator” {

name = “generic_accelerator “

ip_address_type = “IPV4”

enabled = true

}

resource “aws_globalaccelerator_listener” “global_accelerator_listener” {

accelerator_arn = aws_globalaccelerator_accelerator.global_accelerator.id

client_affinity = “SOURCE_IP”

protocol = “TCP”

resource “aws_globalaccelerator_endpoint_group” “global_accelerator_endpoint_primary” {

listener_arn = aws_globalaccelerator_listener.global_accelerator_listener.id

endpoint_group_region = “us-west-2”

health_check_interval_seconds = “30”

health_check_path = “/”

health_check_port = “443”

health_check_protocol = “HTTPS”

threshold_count = “3”

traffic_dial_percentage =”100″

endpoint_configuration {

endpoint_id = var.alb_arn

weight = 255

client_ip_preservation_enabled = “true”

}

}

This will help creating comma(,) separated listener ports in Global accelerator.
Terraform vs CloudFormation — Part 1
By: Amod Kadam
Tags: Compare Tf And Cf, Cloudformation, Terraform, Terraform Vs Cf

I have been using CloudFormation for a while for projects and also cover it as a part of my AWS workshops. In the past couple of years clients have been asking me to cover Terraform as part of AWS workshop !.

This has prompted me to learn and understand Terraform . I have started comparing these toolsets from a practitioners perspective and I thought of putting my experiences in the form of blog posts.

This is the first part of the series where I am going to compare Terraform (TF) and CloudFormation (CF or CFT or CFN).

What is Terraform ?

Terraform (TF) is a tool for building , changing , and versioning infrastructure safely and efficiently

is a tool for , , and safely and efficiently Terraform can manage existing and popular service providers as well as custom in-house solutions

What is CloudFormation?

AWS CloudFormation (CF) is a service that helps you model and set up Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS

High Level Comparison

Let us look at high level comparison points as shown in the following tables.
Running AWS ECS Fargate Spot
By: Adrian Macal
Tags: Fargate, Aws Ecs, Terraform

The AWS ECS offers nice way of running Python containers with ETL work. But spinning hundreds of containers with default settings does not uses cost effective pricing model. In non-essential work you can start bidding spots.

Infrastructure

The minimal changes in the infrastructure should be done in the way how the ECS cluster is provisioned. Here is the version which defines only Fargate Spot as default capacity provider.

Running tasks

The most important change in the code to run ECS task is to not specify launch type at all. I left Fargate there and it still ran, but I could not be able to confirm if the spot was used.

Confirmation

How to be sure if it runs as expected? In the console, in the task you should see new property describing chosen capacity.
Design Uber’s Backend: A Systems Design Walkthrough
By: The Educative Team
Tags: Programming, System Design, Coding Interviews, Uber, Software Development

Uber Backend: System Design

Below, we discuss the system design for this product. If you’re familiar with the SDI question, Designing Yelp, this may look familiar. We will modify that solution for the above use-cases. For Uber, our QuadTree must be adapted for frequent updates.

If we use a dynamic grid solution from our Yelp problem, a few issues arise:

We need to update data structures to reflect that active drivers are reporting their locations every three seconds. To update a driver to a new location, we must find the right grid based on the driver’s previous location.

If the new position does not belong to the current grid, we remove the driver from the current grid and reinsert them to the right grid. Then, if the new grid reaches a maximum limit, we have to repartition it.

We need a quick mechanism to propagate the current location of all the nearby drivers to customers in the area. Our system needs to notify both the driver and passenger on the location of the car throughout the duration of the ride.

For these cases, a QuadTree is not ideal, as a quick update in the tree cannot be guaranteed at the speeds that Uber requires. If we don’t update our QuadTree according to every driver update, it will use old data that does not reflect the current location.

We could keep the most recent driver position in a hash table and update our QuadTree less frequently. We want to guarantee that a driver’s current location is reflected in the QuadTree within 15 seconds. We maintain a hash table that will store the current driver location. We can call it DriverLocationHT .

DriverLocationHT

We need to store DriveID in the hash table, which reflects a driver’s current and previous location. This means that we will need 35 bytes to store each record:

DriverID (3 bytes — 1 million drivers)

Old latitude (8 bytes)

Old longitude (8 bytes)

New latitude (8 bytes)

New longitude (8 bytes) Total = 35 bytes

As we discussed previously, we assume one million drivers, which will require the following memory:

1 million * 35 bytes => 35 MB

Now let’s discuss bandwidth. If we get the DriverID and location, it will require (3+16=>19bytes). This information is received every three seconds from 500,000 daily active drivers, so we receive 9.5MB every three seconds.

To help with scalability, performance, and fault tolerance, we could distribute DriverLocationHT on multiple servers based on the DriverID to randomize distribution. We will refer to the machines holding this info the Driver Location server.

These servers will also do the following:
6 Practical Tips - Typography System Creation
Tags: Design Resources, Design System, UI Design, Visual Design, Typography

There is something necessary in every UI Design — Typography. To speed up your work, the design system must have well-prepared components and a typography system.

In this handy tutorial, I will show you how to improve yours with 6 practical tips:

1. Pick a typeface

Usually, one typeface is enough. If you want to create a more outstanding experience, you may try to mix 2. Use one for headings or title and the second one for smaller sizes like Body, Caption, etc. I love to use good free fonts like Montserrat, Lato, Roboto or Inter. They ensure good readability and modern style.

2. Set the base size

Body font is used to standard text content in your design. It is important to balance good readability with aesthetics. Usually, the perfect size, in this case, is around 16–18 pixels.

3. Create Type Scale

Creating a typography scale is always challenging. There are many techniques that allow you to create a harmonious set. I like to use strong foundations. When you want to create your use sites like type-scale.com or material.io to inspire yourself and build scale quickly.

4. Set Good Line Height

To enhance text readability, it is not enough to have a good font size. Line-height has to fit the font size. For the best results, play with 1.4–1.6 of font size as a line-height. For example if your body font is 16px, the line-height should be around 24px. Try it on yourself, and see the difference in reading comfort.

5. Adjust Tracking

In typography, readability is king. The next factor that helps to enhance it is tracking. Bigger sizes like Headings or Titles require less tracking value. Smaller ones (especially with Upper Case letters) need to have tracking increased.

6. Define meaningful Colors

I always repeat that when it comes to Design System Creation do not use names like “Blue, Green or Yellow.” Make your colors meaningful. For example, give them the following names: “Primary, Secondary, Warning, or Error.” This also applies to font colors. This helps to recognize the right purpose of the element.

Bonus Tips!

Fee free to read the guide about Typography I have written on the blog:

To conclude

Building a Typography System is always challenging — every project requires an individual approach to reflect the brand. However, these simple tips will help you create your bullet-proof design system.