Kubernetes Operator for Beginners — What, Why, How
By: Stefanie Lai
Tags: Operators, Helm, Kudos, Kubernetes, Kubebuilder

Kubernetes Operator for Beginners — What, Why, How

Explain the basic knowledge and means, build an Operator from scratch using Kubebuilder

In Infrastructure Development, we can never bypass Operators in modern Kubernetes solutions.

I have been writing different operators in the past one or two months, with the purpose to reduce companies’ pressure in managing various Kubernetes and GCP resources on the GCP and to inject more automated management methods.

From the company’s perspective, the higher the automation, the lower the maintenance pressure; the stronger the scalability, and the higher the stability.

From the developers’ point of view, application developers can be more assured to focus less on infrastructure and concentrate more on business innovation. Infrastructure developers reduce toil and spare more time to explore interesting things. With Operators, DevOps sleep better at night!😂

from Unsplash @davidclode

Let me unveil Operators’ mystery.

What is an operator

Operator, first proposed by CoreOS engineers in 2016, was to simplify the framework of complex stateful application management. It is a controller that perceives application status and automatically creates, manages, and configures application instances by extending the Kubernetes API.

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop. — from kubernetes.io

The above is the official explanation of Operator from Kubernetes. As long as you understand the two key concepts mentioned, you can understand Operator.

Custom Resource, also known as CRD

Kubernetes is a container orchestration management tool based on Pod, Service, Deployment, Secret, Configmap, and other basic resources. In short, it forms the system architecture we need by integrating the resources in a way as Legos and manages the communication (event), status, expected status (spec) between these resources as well. Thus, only a simple declarative configuration is required for users.

Kubernetes for fun by author

When these basic types fail to meet the demands, customization of the resource type is needed, which is the origin of CRD. By customizing apiVersion, kind, metadata, spec etc., we create our own building blocks, declaratively! As for how to run these ‘building blocks’ in Kubernetes, the Operator is the mechanism behind it.

Control loop

As the core mechanism of the Kubernetes native API, the Controller should be managed by the Controller Manager component inside the system and follow a unified design and development model.

Therefore, CRD, just as the native resources, should be able to be managed by the Kubernetes API and be used by kubectl. That is to put our CRD under the management of the Kubernetes controller. Then what is the Kubernetes controller management pattern?

Operator Pattern

The Operator implementation is based on the controller pattern. It allows application developers to abstract complex distributed application clusters into a customized API object by extending the Kubernetes API object and achieve the basic DevOps by requesting a customized API model. In the Controller, developers can focus on implementing related complex logic encountered in application runtime management.

Why you need an operator

Operator, as a CRD solution, is meaningful. Next, Let’s think about the big picture, enlarging our thoughts.

In principle, Kubernetes is most suitable for a stateless system, because stateless applications support operations such as deployment, migration, and upgrade at any time. Kubernetes’ current ReplicaSets, ReplicationControllers, Services , and other elements are enough to meet stateless applications’ basic requirements for automatic scaling and load balancing between instances.

But with the vigorous development of Kubernetes, it is applied into fields including complex distributed systems, big data, machine learning, etc., facing more and more challenges.

How to keep Kubernetes’s feature while integrating it into stateful systems.

How to ensure the availability of existing instance services when the application system is scaled or upgraded.

How to backup and manage system state and application data.

The answer is CRD, which makes Operator indispensable in the Kubernetes ecosystem.

As an infrastructure developer, I also deal with various Operators every day.

Google Config Sync synchronizes and deploys YAML in Github and maintains final consistency.

Google Config Connector deploys YAML of GCP resources to the cluster.

Prometheus monitors clusters, resources, and uploads metrics to Stackdriver.

Fluentd processes logs to Stackdriver.

There are many other open-source Operators or Operators developed by ourselves. We need them for many reasons.

DRY. Operators define a set of methods for packaging and deploying complex business applications in a Kubernetes cluster and provide a reproducible and standardized solution.

Operators define a set of methods for packaging and deploying complex business applications in a Kubernetes cluster and provide a reproducible and standardized solution. Reliability. Operators also provide a set of monitoring and managing methods at runtime by incorporating business-related operation and maintenance logic into the Operator’s own controller, like a 7*24 operation and maintenance robot. They can monitor the state of the application and theirs watch events in the Kubernetes cluster and process the monitoring events based on the expected final state in a millisecond.

Operators also provide a set of monitoring and managing methods at runtime by incorporating business-related operation and maintenance logic into the Operator’s own controller, like a 7*24 operation and maintenance robot. They can monitor the state of the application and theirs watch events in the Kubernetes cluster and process the monitoring events based on the expected final state in a millisecond. Encapsulation. We need Operators to encapsulate complex business logic and implement a unified external interface (Kubernetes API)

We need Operators to encapsulate complex business logic and implement a unified external interface (Kubernetes API) Declarative. Operators deals with the logic behind the CRD YAML.

In one word

The Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides. — from Kubernetes Operator Pattern

After realizing the role of Operator, let’s see what open-source operators are now available. Operator Hub is where the Kubernetes community hosts all Operators. Here, we can publish our Operator, similar to DockerHub.

How to write an Operator

You can implement an Operator in many ways, with many tools.

Kudo

Helm

Dynamic Operator, Operator SDK, and Kubebuilder

These tools reflect different thinking modes and target different scenarios.

For example, Helm and Kudo both achieve their goals by manipulating YAML files, so the operator implemented by them is essentially a YAML file containing various logics. And then their tools parse this YAML and deploy CRD resources.

Whether it is Kudo or Helm, they are operators themselves. Using them to develop operators is a process of combining different functions under their defined specifications, and many operator templates are offered, like Kube Zookeeper and Kafka.

There are more templates under Helmhub. Moreover, it is easier to publish Operators written by Operator SDK to the community and allow users to perform installation and use through helm because Operator SDK is supportive of Helm.

Dynamic Operator is more complicated. You need to define the CRD characteristic through code to implement the Controller’s reconcile logic and add more complex webhooks, monitoring events, and other logics on its basis. Operator SDK and Kubebuilder are two scaffolding tools that cover some templated code and focus on implementing Operator logic.

Kudo

KUDO allows an application developer to create an operator with declarative measures. — from Kudo doc

To use Kudo, we need to install the Kudo controller in the Cluster. It parses the Operator YAML we wrote and deploy it into the cluster. Meanwhile, we need to install Kudo CLI to perform related management operations.

# install kudo cli on Mac

$brew tap kudobuilder/tap

$brew install kudo-cli # init Kudo in kind cluster

$ kind create cluster --name=kudo-test # init Kudo

$ kubectl kudo init

KUDO_HOME has been configured at /Users/pc/.kudo

$ kubectl api-resources --api-group kudo.dev # check kudo controller installed

$ kubectl get pods -n kudo-system

Install zookeeper via Kudo

# install zk

$ kubectl kudo install zookeeper --instance zk

# verify

$ kubectl get pods

Install Kafka

$ kubectl kudo install kafka --instance=kafka -p BROKER_MEM=1024m # check for information

$ kubectl describe instance kafka | grep -A5 Spec

Of course, we can also define Operators via Kudo in a customized way. Generally, a Kudo Operator contains these elements.

operator.yaml which defines the CRD structure and the operations performed (ADD/DELETE/TOGGLE, etc.)

which defines the CRD structure and the operations performed (ADD/DELETE/TOGGLE, etc.) params.yaml which defines the variables used

which defines the variables used templates.yaml which defines the resource template that applied to the cluster

Kudo itself defines concepts such as Task, Plan, Param, and Template to standardize Operator definitions. The customized Operators should fill in the relevant YAML configuration under these specifications. I won’t go into details here. If interested, you can refer to the Kudo official website to learn more about Kudo architecture and implementation.

Helm

Helm is the best way to find, share, and use software built for Kubernetes. — from helm.sh

Helm emerged for a simple reason. The various native resources of Kubernetes, Pod, Service, Deployment, Secret, etc., are all functionally independent. However, it requires a complex combination of them when we implement a complex component or a specific function.

And the “lazy” developers created a way to standardize operation and simplify usage when they had to repeat the same process under different circumstances. That is the Helm chart.

We can consider Helm, a package manager developed by Deis for Kubernetes, as apt-get/yum under Kubernetes. It is an Operator itself, so you need to install Tiller in the cluster and download the Helm CLI when using it.

Skipping the trivial installation steps and Helm chart usage, let’s directly see how to customize the Helm chart.

$ helm create test-chart

$ ls

Chart.yaml charts templates values.yaml

Chart.yaml , used to describe the chart, including name, description, and version.

, used to describe the chart, including name, description, and version. values.yaml , stores the variables used in the template files in the templates directory

, stores the variables used in the template files in the templates directory Go template file, which is under templates/ . Learn more at Go template info.

Helm CLI supports the release, installation, rollback, and other operations of Helm chart. For more information, please refer to the 👉 Helm.

Dynamic Approach

Dynamic Operator is the freest, most flexible, and most complicated way to implement Operators, which defines the CRD data type by code and implements the Add, Update, Delete operation that serves the Control Pattern resources. The Controller performs resource management by monitoring different Kubernetes events and combining related business logic.

There are two most common methods, both of which implement Operator with Go code.

Operator SDK

Kubebuiler

Both methods use templates to generate code and define data types ( spec, metadata ), and then implement related logic by extending the API with the controller-runtime library . Since it is increasingly harder to distinguish them, the two teams are considering the possibility of merging the two into one. 👉 Discussion

I choose to use Kubebuilder for the following reasons.

Kubebuilder has more mature documentation. kubebuilder.io contains a complete tutorial.

Kubebuilder is maintained by Kubernetes SIG and is more Kubernetes-community-oriented.

Operator SDK is developing towards integrating with Kubebuilder.

Consider that our Operator does not need to be published to the community; otherwise, Operator SDK may be a better choice.

Implement an Operator through Kubebuilder

To implement an Operator, “do one thing and do it well” is a good rule of thumb, my interpretation of which is three one-one-mapping.

Operator build process

There are only four steps to implement an Operator.

Define the CRD. To be more specific, define Spec/Status/Metadata

Generate template code with Kubebuilder tool

Implement the reconcile logic in the generated Controller

Test

Let’s walk through the process step by step.

Suppose we have such a case.

Implement a function to automatically generate the user’s ServiceAccount and ClusterRolebinding containing configuration according to the user’s information.

For instance, we have a user Jenny and she has a GSA account. We want to use an Operator to connect the GSA to a certain ClusterRole. So Jenny has the correct cluster permissions. Below are the resources we want the Operator to create/apply to the cluster automatically

Put it simply, firstly, user information is dynamic and obtained through Http Rest or gRPC interface. Secondly, we need to create resources for new users and delete information for logged-out users.

Initialization

After initializing the Kubebuilder tool, we get the following directory structure.

$ kubebuilder init --domain company.org --license apache2 --owner pc

The directory structure after the init command

Kubebuilder init

config dir contains many boilerplate YAML files, such as RBAC, webhook, cert, etc.

contains many boilerplate YAML files, such as RBAC, webhook, cert, etc. main.go is the entrance of the whole Operator.

is the entrance of the whole Operator. Makefile contains many Go command wrapper.

Now we can create the API and pass flags like group, version, kind. Usually, people consider GKV as the unique identity of the CRD.

$ kubebuilder create api --group identity --version v1 --kind UserIdentity

Choose yes to create both Resource and Controller.

create Output

After running create command, you will find there are two directories (api/ and controller/) created.

API dir

controller dir

These two places are where the magic happens.

Define CRD

useridentity_types.go is the place we define our CRD fields. We can find out it already has many default fields set, like UserIdentitySpec , UserIdentityStatus , and meta type fields.

In our example, the only thing we need the operator to carry is the ClusterRole ref, which the user gets from a gRPC call or HTTP request, or even a PubSub topic subscribe.

So All I need to do is to put the field into the UserIdentitySpec .

type UserIdentitySpec struct {

// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster

// Important: Run "make" to regenerate code after modifying this file



// roleRef is the target ClusterRole reference

roleRef string `json:"roleRef,omitempty"`

}

After setting up the CRD fields, we can also change the YAML sample of the CRD in config/smaples/identity_b1_useridentity.yaml to below:

apiVersion: identity.company.org/v1

kind: UserIdentity

metadata:

name: useridentity-sample

spec:

# Add fields here

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: event-view

Define Controller

The easiest way to implement a controller is the method of implementing reconcile.

To be simple, we will skip the getting user name part, and focus on the interaction between the controller runtime API.

The approximate reconciliation logic in controllers/useridentity_controller.go is as follows.

Test

After completing all the above, we can write suite_test to test our code. Kubebuilder itself integrates the ginkgo framework, uses the BDD pattern to test the cases, and completes the test with the gomega framework.

A simple suite_test case is similar:

When("A ClusterRoleBinding is created", func() {



It("Should find a GroupRoleBinding", func() {

Eventually(func() bool {

fetched := &rbacv1.ClusterRoleBinding{}

_ = k8sClient.Get(context.TODO(), req, fetched)

return fetched != nil

}, 10, 1).Should(Succeed())

})

})

You can run the test by using commands in the Makefile

make test

The next step is to deploy the Operator to the local kind cluster for testing.

make install kubectl apply -f config/samples/useridentity.yaml

Use the sample YAML, and kubectl apply it to the cluster.

Finally, we can verify by kubectl get clusterrolebinding.

At this point, our Operator has been initially implemented.

Is Operator Necessary

In Kubecon NA 2020, many talks were around Operators' implementation and way of thinking. And some community members even mentioned the necessity of Operators as a continuous delivery model for Kubernetes in Keynotes.

However, some people hold a different opinion. Joe Thompson from HashiCorp published a talk on “Stop Writing Operators” and put forward several points that I agree on.

I agree Operator is the ultimate means to solve the problem, not the only one or the primary one.

Do not indulge in automation-If a task only takes 10 minutes a month, and then it is not worth spending a week or more writing an Operator.

Meanwhile, if you already have the current Helm chart or Kudo template, never waste time re-creating the wheels yourself. If not, see if you can implement it through the Helm chart other than using the Operator SDK or Kubebuilder to implement it anymore because Declarative’s YAML is always simpler than Go code.

We encountered many problems during the implementation process, including Kubebuilder itself, issues with SIG upstream tools. The time spent on solving problems far exceeded the estimated.

If possible, maybe I will keep our “ugly” bash script instead of customizing the Operator.

To Sum Up

This article is a preliminary summary of my research on Operators in the past few months, which records the course from understanding Operators to calling various operator implementation tools, from trying the tools to solving the problems with Kubebuilder.

Actually, I encountered more problems in Kubebuilder coding that I will summarize in the follow-up articles. Although implementing Operators is not easy, it is indeed a way to understand the Kubernetes controller-runtime life cycle better.

The code in the article is on Github.

Thanks for reading!