Continuous GitOps, the way to do DevOps in Kubernetes
By: Arun Ramakani
Tags: K8s, Gitops, Continuous Delivery, Kubernetes, DevOps

Continuous GitOps, the way to do DevOps in Kubernetes

Continuous GitOps, the new age DevOps practice to increase the delivery velocity by achieving an end to end “Git source of truth” with Zero manual changes into the Kubernetes cluster 🏄 Arun Ramakani Follow Jan 19, 2020 · 8 min read

It’s been 10 years now from the time, when we first heard of the word Continuous Delivery. It’s Humble Jez and Farley David talked about Continuous Delivery during 2010 through their book “Continuous Delivery: Reliable Software Releases Through Build, Test and Deployment Automation”. In the last decade, Continuous Delivery has changed the way we do Software Releases. Now with a new set of tools evolving around the Kubernetes ecosystem, we are taking yet another leap in the Continuous Delivery journey. These tools revolve around the concept of Continuous GitOps. This blog is an attempt to demystify the Why? What? and How? of “Continuous GitOps”.

What is Continuous GitOp?

Continuous GitOps is a concept, where we look at having an end to end description of the software into the Git and then try to make the cluster state in sync with the Git repository continuously. There are two keywords here that will make a difference.

Description Representation of Software: With Kubernetes, the relationship between the application and the underlying infrastructure is purely declarative. You ask for what do you want from the infrastructure with declarative (YAML’s). The implementation details of these YAML’s are abstracted away by the underlying Kubernetes cluster with Controllers, Schedulers, KubeDNS, KubeProxy, Operators, etc., This enables us to move away from the traditional “Infrastructure as Code” to “Infrastructure as Data”. More on this at GitHub. The key here is, every declarative tag that you need for an application (from Application Developer / Application Operator / Cluster Operator) is enriched into the YAML in the Continuous Delivery pipeline and finally push into the GitOps Repository. And should have zero tolerance towards any manual YAML changes in the cluster. Continuous Sync: The continuous sync means, continuously looking into the git repositories to reflect any state change into the Kubernetes cluster. This is a powerful idea that came from a tool called Flux. The Flux attempted to move the deployment automation from outside the Kubernetes cluster, into the cluster using Kubernetes Operators.

Continuous GitOps is an intersection of four major components working together creating the magic

Git Repository: The source code repository that is storing our application’s declarative definition as YAML’s. Kubernetes Cluster: The underlying cluster where we are deploying our applications. Sync Agent: The Kubernetes operator extension which is doing the job of continuously syncing the Git Repository and the application state into the cluster. CD Pipeline: The Continuous Deployment pipeline that orchestrates the whole flow.

The reference architecture on how these components work together to create a Continuous GitOps flow is given below.

The YAML creation and enrichment are divided into three parts between the Application Developer, Application Operator and Cluster Operator in the above reference architecture. Depending on your organization team structure, cluster multi-tenancy, and governance need, you may choose to do it in one or two steps. We will look into this more in “Managing Declarative YAML’s” section down below.

Now let’s have a look at, Why Continuous GitOps?

Why Continuous GitOps?

Continuous GitOps add value in many dimensions. Let’s look at some of the critical ones,

Application Delivery Velocity:

Continuous GitOps will increase the product delivery velocity in the following ways,

Confident rollout with the ability to compare the final full YAML and the cluster state. This can also act as a decision guide for approving the releases. Easy rollout process with automated blue-green deployment, powered by application metrics from Prometheus. Automated container image update based on policy. For example, Istio sidecar minor versions releases are backward compatible and can be updated automatically. GitOps will make Operations, developer-centric providing a low friction development experience. The application team can take over some operation work and the operation team can concentrate on the platform. GitOps repository acts as a way to bypass the full continuous deployment flow for urgent release.

End to End Automation:

In the GitOps approach, everything that concerns an Application Developer, Application Operator and Cluster Operator are embedded into the YAML through git enabling an end to end automation.

Security, Audit & Compliance:

The policy of Zero manual changes into the cluster will hugely increase the cluster security and support any compliance requirement. Since everything in the cluster is in git, we will have a full audit log of what is happening to the cluster.

Cluster Observability:

With a full audit log, we can easily get the list of changes happened in the cluster to assist any issue debugging.

Separation Of Concerns & Easy Migration:

Continuous GitOps will enable segregation of concerns between the application developer, application operator and cluster operator concerns. The dependencies from these teams are injected into git in a declarative way as dependency injection. This will ease our migration of underlying k8s cluster, governance policy or cross-cutting tools.

Read more about an end to end DevOps workflow Workflow for Kubernetes DevOps.

How can we adopt Continuous GitOps?

We will look into the following four different aspects, which will help us get this going

GitOps workflows to implement

Managing declarative YAML’s

The tools

Where to start?

GitOps workflows to implement

The following three workflows are the popular workflows to adopt when we start with GitOps

Workflow 1: Standard GitOps Flow

This is the standard GitOps Workflow where we push the application YAML description into the GitOps repository and a GitOps agent will Auto Sync the state change.

Workflow 2: Auto-Image Update

In this workflow, the GitOps agent will auto-update the new version of the container image from the container image registry based on a specified policy. For example, we can set up a policy like, if there is a minor version change in the image we can auto-update as they are backward compatible.

Workflow 3: Automated Canary Deployment

This is a powerful workflow where we can automate Canary Deployments. With this, we gradually shift traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pods health with Prometheus. Based on analysis of the KPIs a canary is promoted or aborted

Managing Declarative YAML’s

Let’s say we have an e-commerce cart application and the full application definition is a mix of the below

The Application Image. The YAML for Pod, Deployment, Services, Volume, and ConfigMap. Few sealed secrets for Database communication. Tag to use Istio service mesh as the default cross-cutting sidecar as a Cluster Governance Policies. Environment Governance policy like one replication in staging and 3 replication in production. Tag’s to add node affinity and toleration's for high availability node scheduling. Network security policy YAML based on pod labels.

To build a final application YAML, we need input from the application developer, application operator, and cluster operator. Depending on your organization team structure, cluster multi-tenancy, and governance complexities, all these activities may be performed by a single to three different teams. Let’s look at this more in detail.

Assume that you are a small organization and managing hand full pod’s, you will have small

The below flow represents how we can build an orchestrated Continuous Deployment Automation pipeline.

This way of segregating the concerns is heavily influenced by the Open Application Model, which attempts to provide a framework for cloud-native application development.

Open Application Model [OAM] describes a model where

Developers are responsible for defining application components.

are responsible for defining application components. Application operators are responsible for creating instances of those components and assigning them application configurations.

are responsible for creating instances of those components and assigning them application configurations. Infrastructure operators are responsible for declaring, installing, and maintaining the underlying services that are available on the platform.

By applying the OAM framework, the below table attempts to segregate the YAML contribution responsibilities. This is may change based on how your organization is structured and the type of Kubernetes cluster you choose.

The Tools

If you are convinced with the idea of GitOps, your next step is to identify the tools required. There are many tools that can support us to achieve a different aspect of GitOps. Let's look at some of these tools and their usage

Git: This is the base of everything that we do with GitOps to store our YAML artifacts.

Helm & Kustomize: This is a powerful combination that can help us generate declarative YAML manifest. We can package the application and its dependencies with Helm. Then Kustomize will help us to customize and patch the YAML files leaving the original YAML untouched. Read more on this at Helm Is Not Enough, You Also Need Kustomize. If you are looking for some other alternative tools to do this Read.

Argo CD: It’s a GitOps continuous delivery tool, that can act as an agent to sync the changes from the GitOps repository into the Kubernetes cluster. (Argo CD)

Flux: It’s another GitOps continuous delivery tool, that can act as an agent to sync the changes from the GitOps repository into the Kubernetes cluster. (Flux)

Flagger: This tool works well with flux to deliver canary deployments with GitOps (Flagger)

Where to Start?

If you are starting a greenfield project, it’s a bit easy to adopt GitOps from the beginning. All we have do is to choose our tools for CI/CD, declarative YAML management and GitOps Agent to get it going.

If you are looking for implementing GitOps in a brownfield application, below points may guide you

You can pick one application at a time, and use the success story to spread wider adoption. When you select the first application, choose the one that changes more frequently. This will help us to build some solid metrics for the success story. Choose an application that breaks frequently. With GitOps, these applications should break less frequently and when it breaks you should have better observability. Prefer business application over operational cross-cutting concerns like Istio, RBAC Integration, etc. These operational cross-cutting concerns are complex and not a low hanging fruit. If required introduce dummy manual approval step, till you get confidence.

Let’s walk through GitOps step by step in an upcoming article.