Chapter 5. Developing Operators 5.1. About the Operator SDK The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators , in an effective, automated, and scalable way. Operators take advantage of Kubernetes extensibility to deliver the automation advantages of cloud services, like provisioning, scaling, and backup and restore, while being able to run anywhere that Kubernetes can run. Operators make it easy to manage complex, stateful applications on top of Kubernetes. However, writing an Operator today can be difficult because of challenges such as using low-level APIs, writing boilerplate, and a lack of modularity, which leads to duplication. The Operator SDK, a component of the Operator Framework, provides a command-line interface (CLI) tool that Operator developers can use to build, test, and deploy an Operator. Why use the Operator SDK? The Operator SDK simplifies this process of building Kubernetes-native applications, which can require deep, application-specific operational knowledge. The Operator SDK not only lowers that barrier, but it also helps reduce the amount of boilerplate code required for many common management capabilities, such as metering or monitoring. The Operator SDK is a framework that uses the controller-runtime library to make writing Operators easier by providing the following features: High-level APIs and abstractions to write the operational logic more intuitively Tools for scaffolding and code generation to quickly bootstrap a new project Integration with Operator Lifecycle Manager (OLM) to streamline packaging, installing, and running Operators on a cluster Extensions to cover common Operator use cases Metrics set up automatically in any generated Go-based Operator for use on clusters where the Prometheus Operator is deployed Operator authors with cluster administrator access to a Kubernetes-based cluster (such as OpenShift Container Platform) can use the Operator SDK CLI to develop their own Operators based on Go, Ansible, or Helm. Kubebuilder is embedded into the Operator SDK as the scaffolding solution for Go-based Operators, which means existing Kubebuilder projects can be used as is with the Operator SDK and continue to work. Note OpenShift Container Platform 4.10 supports Operator SDK v1.16.0. 5.1.1. What are Operators? For an overview about basic Operator concepts and terminology, see Understanding Operators . 5.1.2. Development workflow The Operator SDK provides the following workflow to develop a new Operator: Create an Operator project by using the Operator SDK command-line interface (CLI). Define new resource APIs by adding custom resource definitions (CRDs). Specify resources to watch by using the Operator SDK API. Define the Operator reconciling logic in a designated handler and use the Operator SDK API to interact with resources. Use the Operator SDK CLI to build and generate the Operator deployment manifests. Figure 5.1. Operator SDK workflow At a high level, an Operator that uses the Operator SDK processes events for watched resources in an Operator author-defined handler and takes actions to reconcile the state of the application. 5.1.3. Additional resources Certified Operator Build Guide 5.2. Installing the Operator SDK CLI The Operator SDK provides a command-line interface (CLI) tool that Operator developers can use to build, test, and deploy an Operator. You can install the Operator SDK CLI on your workstation so that you are prepared to start authoring your own Operators. Operator authors with cluster administrator access to a Kubernetes-based cluster, such as OpenShift Container Platform, can use the Operator SDK CLI to develop their own Operators based on Go, Ansible, or Helm. Kubebuilder is embedded into the Operator SDK as the scaffolding solution for Go-based Operators, which means existing Kubebuilder projects can be used as is with the Operator SDK and continue to work. Note OpenShift Container Platform 4.10 supports Operator SDK v1.16.0. 5.2.1. Installing the Operator SDK CLI You can install the OpenShift SDK CLI tool on Linux. Prerequisites Go v1.16+ docker v17.03+, podman v1.9.3+, or buildah v1.7+ Procedure Navigate to the OpenShift mirror site . From the latest 4.10 directory, download the latest version of the tarball for Linux. Unpack the archive: USD tar xvf operator-sdk-v1.16.0-ocp-linux-x86_64.tar.gz Make the file executable: USD chmod +x operator-sdk Move the extracted operator-sdk binary to a directory that is on your PATH . Tip To check your PATH : USD echo USDPATH USD sudo mv ./operator-sdk /usr/local/bin/operator-sdk Verification After you install the Operator SDK CLI, verify that it is available: USD operator-sdk version Example output operator-sdk version: "v1.16.0-ocp", ... 5.3. Updating projects for newer Operator SDK versions OpenShift Container Platform 4.10 supports Operator SDK v1.16.0. If you already have the v1.10.1 CLI installed on your workstation, you can update the CLI to v1.16.0 by installing the latest version . However, to ensure your existing Operator projects maintain compatibility with Operator SDK v1.16.0, update steps are required for the associated breaking changes introduced since v1.10.1. You must perform the update steps manually in any of your Operator projects that were previously created or maintained with v1.10.1. 5.3.1. Updating projects for Operator SDK v1.16.0 The following procedure updates an existing Operator project for compatibility with v1.16.0. Important Operator SDK v1.16.0 supports Kubernetes 1.22. Many deprecated v1beta1 APIs were removed in Kubernetes 1.22, including sigs.k8s.io/controller-runtime v0.10.0 and controller-gen v0.7 . Updating projects to Kubernetes 1.22 is a breaking change if you need to scaffold v1beta1 APIs for custom resource definitions (CRDs) or webhooks to publish your project into older cluster versions. See Validating bundle manifests for APIs removed from Kubernetes 1.22 and Beta APIs removed from Kubernetes 1.22 for more information about changes introduced in Kubernetes 1.22. Prerequisites Operator SDK v1.16.0 installed. An Operator project created or maintained with Operator SDK v1.10.1. Procedure Add the protocol field in the config/default/manager_auth_proxy_patch.yaml and config/rbac/auth_proxy_service.yaml files: ... ports: - containerPort: 8443 + protocol: TCP name: https Make the following changes to the config/manager/manager.yaml file: Increase the CPU and memory resource limits: resources: limits: - cpu: 100m - memory: 30Mi + cpu: 200m + memory: 100Mi Add an annotation to specify the default container manager: ... template: metadata: annotations: kubectl.kubernetes.io/default-container: manager ... Add PHONY targets to all of the targets in your Makefile file. For Go-based Operator projects, make the following changes: Install the setup-envtest binary. Change your go.mod file to update the dependencies: k8s.io/api v0.22.1 k8s.io/apimachinery v0.22.1 k8s.io/client-go v0.22.1 sigs.k8s.io/controller-runtime v0.10.0 Run the go mod tidy command to download the dependencies: USD go mod tidy Make the following changes to your Makefile file: ... + ENVTEST_K8S_VERSION = 1.22 test: manifests generate fmt vet envtest ## Run tests. - go test ./... -coverprofile cover.out + KUBEBUILDER_ASSETS="USD(shell USD(ENVTEST) use USD(ENVTEST_K8S_VERSION) -p path)" go test ./... -coverprofile cover.out ... - USD(CONTROLLER_GEN) USD(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases + USD(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases ... # Produce CRDs that work back to Kubernetes 1.11 (no version conversion) - CRD_OPTIONS ?= "crd:trivialVersions=true,preserveUnknownFields=false" ... - admissionReviewVersions={v1,v1beta1} + admissionReviewVersions=v1 ... + ifndef ignore-not-found + ignore-not-found = false + endif ##@ Deployment ... - sh kubectl delete -f - + sh kubectl delete --ignore-not-found=USD(ignore-not-found) -f - Run the make manifest command to generate your manifests with the updated version of Kubernetes: USD make manifest For Ansible-based Operator projects, make the following changes: Change your requirements.yml file to include the following: Replace the community.kubernetes collection with the kubernetes.core collection: ... - name: kubernetes.core version: "2.2.0" ... Update the operator_sdk.util utility from version 0.2.0 to 0.3.1 : ... - name: operator_sdk.util version: "0.3.1" Verify the default resource limits in the config/manager/manager.yaml file: ... # TODO(user): Configure the resources accordingly based on the project requirements. # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ resources: limits: cpu: 500m memory: 768Mi requests: cpu: 10m memory: 256Mi Important Operator SDK scaffolds these values as a reasonable default setting. Operator authors should set and optimize resource limits based on the requirements of their project. Optional: Make the following changes if you want to run your Ansible-based Operator locally by using the make run command: Change the run target in the Makefile file: ANSIBLE_ROLES_PATH="USD(ANSIBLE_ROLES_PATH):USD(shell pwd)/roles" USD(ANSIBLE_OPERATOR) run Update the local version of ansible-runner to 2.0.2 or later. Important As of version 2.0, the ansible-runner tool includes changes in the command signature that are not compatible with earlier versions. 5.3.2. Additional resources Migrating package manifest projects to bundle format Upgrading projects for Operator SDK v1.10.1 Upgrading projects for Operator SDK v1.8.0 5.4. Go-based Operators 5.4.1. Getting started with Operator SDK for Go-based Operators To demonstrate the basics of setting up and running a Go-based Operator using tools and libraries provided by the Operator SDK, Operator developers can build an example Go-based Operator for Memcached, a distributed key-value store, and deploy it to a cluster. 5.4.1.1. Prerequisites Operator SDK CLI installed OpenShift CLI ( oc ) v4.10+ installed Logged into an OpenShift Container Platform 4.10 cluster with oc with an account that has cluster-admin permissions To allow the cluster to pull the image, the repository where you push your image must be set as public, or you must configure an image pull secret 5.4.1.2. Creating and deploying Go-based Operators You can build and deploy a simple Go-based Operator for Memcached by using the Operator SDK. Procedure Create a project. Create your project directory: USD mkdir memcached-operator Change into the project directory: USD cd memcached-operator Run the operator-sdk init command to initialize the project: USD operator-sdk init \ --domain=example.com \ --repo=github.com/example-inc/memcached-operator The command uses the Go plugin by default. Create an API. Create a simple Memcached API: USD operator-sdk create api \ --resource=true \ --controller=true \ --group cache \ --version v1 \ --kind Memcached Build and push the Operator image. Use the default Makefile targets to build and push your Operator. Set IMG with a pull spec for your image that uses a registry you can push to: USD make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag> Run the Operator. Install the CRD: USD make install Deploy the project to the cluster. Set IMG to the image that you pushed: USD make deploy IMG=<registry>/<user>/<image_name>:<tag> Create a sample custom resource (CR). Create a sample CR: USD oc apply -f config/samples/cache_v1_memcached.yaml \ -n memcached-operator-system Watch for the CR to reconcile the Operator: USD oc logs deployment.apps/memcached-operator-controller-manager \ -c manager \ -n memcached-operator-system Delete a CR Delete a CR by running the following command: USD oc delete -f config/samples/cache_v1_memcached -n memcached-operator-system Clean up. Run the following command to clean up the resources that have been created as part of this procedure: USD make undeploy 5.4.1.3. steps See Operator SDK tutorial for Go-based Operators for a more in-depth walkthrough on building a Go-based Operator. 5.4.2. Operator SDK tutorial for Go-based Operators Operator developers can take advantage of Go programming language support in the Operator SDK to build an example Go-based Operator for Memcached, a distributed key-value store, and manage its lifecycle. This process is accomplished using two centerpieces of the Operator Framework: Operator SDK The operator-sdk CLI tool and controller-runtime library API Operator Lifecycle Manager (OLM) Installation, upgrade, and role-based access control (RBAC) of Operators on a cluster Note This tutorial goes into greater detail than Getting started with Operator SDK for Go-based Operators . 5.4.2.1. Prerequisites Operator SDK CLI installed OpenShift CLI ( oc ) v4.10+ installed Logged into an OpenShift Container Platform 4.10 cluster with oc with an account that has cluster-admin permissions To allow the cluster to pull the image, the repository where you push your image must be set as public, or you must configure an image pull secret 5.4.2.2. Creating a project Use the Operator SDK CLI to create a project called memcached-operator . Procedure Create a directory for the project: USD mkdir -p USDHOME/projects/memcached-operator Change to the directory: USD cd USDHOME/projects/memcached-operator Activate support for Go modules: USD export GO111MODULE=on Run the operator-sdk init command to initialize the project: USD operator-sdk init \ --domain=example.com \ --repo=github.com/example-inc/memcached-operator Note The operator-sdk init command uses the Go plugin by default. The operator-sdk init command generates a go.mod file to be used with Go modules . The --repo flag is required when creating a project outside of USDGOPATH/src/ , because generated files require a valid module path. 5.4.2.2.1. PROJECT file Among the files generated by the operator-sdk init command is a Kubebuilder PROJECT file. Subsequent operator-sdk commands, as well as help output, that are run from the project root read this file and are aware that the project type is Go. For example: domain: example.com layout: - go.kubebuilder.io/v3 projectName: memcached-operator repo: github.com/example-inc/memcached-operator version: "3" plugins: manifests.sdk.operatorframework.io/v2: {} scorecard.sdk.operatorframework.io/v2: {} sdk.x-openshift.io/v1: {} 5.4.2.2.2. About the Manager The main program for the Operator is the main.go file, which initializes and runs the Manager . The Manager automatically registers the Scheme for all custom resource (CR) API definitions and sets up and runs controllers and webhooks. The Manager can restrict the namespace that all controllers watch for resources: mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: namespace}) By default, the Manager watches the namespace where the Operator runs. To watch all namespaces, you can leave the namespace option empty: mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: ""}) You can also use the MultiNamespacedCacheBuilder function to watch a specific set of namespaces: var namespaces []string 1 mgr, err := ctrl.NewManager(cfg, manager.Options{ 2 NewCache: cache.MultiNamespacedCacheBuilder(namespaces), }) 1 List of namespaces. 2 Creates a Cmd struct to provide shared dependencies and start components. 5.4.2.2.3. About multi-group APIs Before you create an API and controller, consider whether your Operator requires multiple API groups. This tutorial covers the default case of a single group API, but to change the layout of your project to support multi-group APIs, you can run the following command: USD operator-sdk edit --multigroup=true This command updates the PROJECT file, which should look like the following example: domain: example.com layout: go.kubebuilder.io/v3 multigroup: true ... For multi-group projects, the API Go type files are created in the apis/<group>/<version>/ directory, and the controllers are created in the controllers/<group>/ directory. The Dockerfile is then updated accordingly. Additional resource For more details on migrating to a multi-group project, see the Kubebuilder documentation . 5.4.2.3. Creating an API and controller Use the Operator SDK CLI to create a custom resource definition (CRD) API and controller. Procedure Run the following command to create an API with group cache , version, v1 , and kind Memcached : USD operator-sdk create api \ --group=cache \ --version=v1 \ --kind=Memcached When prompted, enter y for creating both the resource and controller: Create Resource [y/n] y Create Controller [y/n] y Example output Writing scaffold for you to edit... api/v1/memcached_types.go controllers/memcached_controller.go ... This process generates the Memcached resource API at api/v1/memcached_types.go and the controller at controllers/memcached_controller.go . 5.4.2.3.1. Defining the API Define the API for the Memcached custom resource (CR). Procedure Modify the Go type definitions at api/v1/memcached_types.go to have the following spec and status : // MemcachedSpec defines the desired state of Memcached type MemcachedSpec struct { // +kubebuilder:validation:Minimum=0 // Size is the size of the memcached deployment Size int32 `json:"size"` } // MemcachedStatus defines the observed state of Memcached type MemcachedStatus struct { // Nodes are the names of the memcached pods Nodes []string `json:"nodes"` } Update the generated code for the resource type: USD make generate Tip After you modify a *_types.go file, you must run the make generate command to update the generated code for that resource type. The above Makefile target invokes the controller-gen utility to update the api/v1/zz_generated.deepcopy.go file. This ensures your API Go type definitions implement the runtime.Object interface that all Kind types must implement. 5.4.2.3.2. Generating CRD manifests After the API is defined with spec and status fields and custom resource definition (CRD) validation markers, you can generate CRD manifests. Procedure Run the following command to generate and update CRD manifests: USD make manifests This Makefile target invokes the controller-gen utility to generate the CRD manifests in the config/crd/bases/cache.example.com_memcacheds.yaml file. 5.4.2.3.2.1. About OpenAPI validation OpenAPIv3 schemas are added to CRD manifests in the spec.validation block when the manifests are generated. This validation block allows Kubernetes to validate the properties in a Memcached custom resource (CR) when it is created or updated. Markers, or annotations, are available to configure validations for your API. These markers always have a +kubebuilder:validation prefix. Additional resources For more details on the usage of markers in API code, see the following Kubebuilder documentation: CRD generation Markers List of OpenAPIv3 validation markers For more details about OpenAPIv3 validation schemas in CRDs, see the Kubernetes documentation . 5.4.2.4. Implementing the controller After creating a new API and controller, you can implement the controller logic. Procedure For this example, replace the generated controller file controllers/memcached_controller.go with following example implementation: Example 5.1. Example memcached_controller.go /*

Commands:
tar xvf operator-sdk-v1.16.0-ocp-linux-x86_64.tar.gz
chmod +x operator-sdk
echo USDPATH
sudo mv ./operator-sdk /usr/local/bin/operator-sdk
operator-sdk version
operator-sdk version: "v1.16.0-ocp",
ports: - containerPort: 8443 + protocol: TCP name: https
resources: limits: - cpu: 100m - memory: 30Mi + cpu: 200m + memory: 100Mi
template: metadata: annotations: kubectl.kubernetes.io/default-container: manager
k8s.io/api v0.22.1 k8s.io/apimachinery v0.22.1 k8s.io/client-go v0.22.1 sigs.k8s.io/controller-runtime v0.10.0
go mod tidy
+ ENVTEST_K8S_VERSION = 1.22 test: manifests generate fmt vet envtest ## Run tests. - go test ./... -coverprofile cover.out + KUBEBUILDER_ASSETS="USD(shell USD(ENVTEST) use USD(ENVTEST_K8S_VERSION) -p path)" go test ./... -coverprofile cover.out - USD(CONTROLLER_GEN) USD(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases + USD(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases Produce CRDs that work back to Kubernetes 1.11 (no version conversion) - CRD_OPTIONS ?= "crd:trivialVersions=true,preserveUnknownFields=false" - admissionReviewVersions={v1,v1beta1} + admissionReviewVersions=v1 + ifndef ignore-not-found + ignore-not-found = false + endif ##@ Deployment - sh kubectl delete -f - + sh kubectl delete --ignore-not-found=USD(ignore-not-found) -f -
make manifest
- name: kubernetes.core version: "2.2.0"
- name: operator_sdk.util version: "0.3.1"
# TODO(user): Configure the resources accordingly based on the project requirements. # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ resources: limits: cpu: 500m memory: 768Mi requests: cpu: 10m memory: 256Mi
ANSIBLE_ROLES_PATH="USD(ANSIBLE_ROLES_PATH):USD(shell pwd)/roles" USD(ANSIBLE_OPERATOR) run
mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain=example.com --repo=github.com/example-inc/memcached-operator
operator-sdk create api --resource=true --controller=true --group cache --version v1 --kind Memcached
make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>
make install
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc apply -f config/samples/cache_v1_memcached.yaml -n memcached-operator-system
oc logs deployment.apps/memcached-operator-controller-manager -c manager -n memcached-operator-system
oc delete -f config/samples/cache_v1_memcached -n memcached-operator-system
make undeploy
mkdir -p USDHOME/projects/memcached-operator
cd USDHOME/projects/memcached-operator
export GO111MODULE=on
operator-sdk init --domain=example.com --repo=github.com/example-inc/memcached-operator
domain: example.com layout: - go.kubebuilder.io/v3 projectName: memcached-operator repo: github.com/example-inc/memcached-operator version: "3" plugins: manifests.sdk.operatorframework.io/v2: {} scorecard.sdk.operatorframework.io/v2: {} sdk.x-openshift.io/v1: {}
mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: namespace})
mgr, err := ctrl.NewManager(cfg, manager.Options{Namespace: ""})
var namespaces []string 1 mgr, err := ctrl.NewManager(cfg, manager.Options{ 2 NewCache: cache.MultiNamespacedCacheBuilder(namespaces), })
operator-sdk edit --multigroup=true
domain: example.com layout: go.kubebuilder.io/v3 multigroup: true
operator-sdk create api --group=cache --version=v1 --kind=Memcached
Create Resource [y/n] y Create Controller [y/n] y
Writing scaffold for you to edit api/v1/memcached_types.go controllers/memcached_controller.go
// MemcachedSpec defines the desired state of Memcached type MemcachedSpec struct { // +kubebuilder:validation:Minimum=0 // Size is the size of the memcached deployment Size int32 `json:"size"` } // MemcachedStatus defines the observed state of Memcached type MemcachedStatus struct { // Nodes are the names of the memcached pods Nodes []string `json:"nodes"` }
make generate
make manifests
/* Copyright 2020. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package controllers import ( appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "reflect" "context" "github.com/go-logr/logr" "k8s.io/apimachinery/pkg/runtime" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" ctrllog "sigs.k8s.io/controller-runtime/pkg/log" cachev1 "github.com/example-inc/memcached-operator/api/v1" ) // MemcachedReconciler reconciles a Memcached object type MemcachedReconciler struct { client.Client Log logr.Logger Scheme *runtime.Scheme } // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list; // Reconcile is part of the main kubernetes reconciliation loop which aims to // move the current state of the cluster closer to the desired state. // TODO(user): Modify the Reconcile function to compare the state specified by // the Memcached object against the actual cluster state, and then // perform operations to make the cluster state reflect the state specified by // the user. // // For more details, check Reconcile and its Result here: // - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.7.0/pkg/reconcile func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { //log := r.Log.WithValues("memcached", req.NamespacedName) log := ctrllog.FromContext(ctx) // Fetch the Memcached instance memcached := &cachev1.Memcached{} err := r.Get(ctx, req.NamespacedName, memcached) if err != nil { if errors.IsNotFound(err) { // Request object not found, could have been deleted after reconcile request. // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers. // Return and don't requeue log.Info("Memcached resource not found. Ignoring since object must be deleted") return ctrl.Result{}, nil } // Error reading the object - requeue the request. log.Error(err, "Failed to get Memcached") return ctrl.Result{}, err } // Check if the deployment already exists, if not create a new one found := &appsv1.Deployment{} err = r.Get(ctx, types.NamespacedName{Name: memcached.Name, Namespace: memcached.Namespace}, found) if err != nil && errors.IsNotFound(err) { // Define a new deployment dep := r.deploymentForMemcached(memcached) log.Info("Creating a new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name) err = r.Create(ctx, dep) if err != nil { log.Error(err, "Failed to create new Deployment", "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name) return ctrl.Result{}, err } // Deployment created successfully - return and requeue return ctrl.Result{Requeue: true}, nil } else if err != nil { log.Error(err, "Failed to get Deployment") return ctrl.Result{}, err } // Ensure the deployment size is the same as the spec size := memcached.Spec.Size if *found.Spec.Replicas != size { found.Spec.Replicas = &size err = r.Update(ctx, found) if err != nil { log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name) return ctrl.Result{}, err } // Spec updated - return and requeue return ctrl.Result{Requeue: true}, nil } // Update the Memcached status with the pod names // List the pods for this memcached's deployment podList := &corev1.PodList{} listOpts := []client.ListOption{ client.InNamespace(memcached.Namespace), client.MatchingLabels(labelsForMemcached(memcached.Name)), } if err = r.List(ctx, podList, listOpts...); err != nil { log.Error(err, "Failed to list pods", "Memcached.Namespace", memcached.Namespace, "Memcached.Name", memcached.Name) return ctrl.Result{}, err } podNames := getPodNames(podList.Items) // Update status.Nodes if needed if !reflect.DeepEqual(podNames, memcached.Status.Nodes) { memcached.Status.Nodes = podNames err := r.Status().Update(ctx, memcached) if err != nil { log.Error(err, "Failed to update Memcached status") return ctrl.Result{}, err } } return ctrl.Result{}, nil } // deploymentForMemcached returns a memcached Deployment object func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1.Memcached) *appsv1.Deployment { ls := labelsForMemcached(m.Name) replicas := m.Spec.Size dep := &appsv1.Deployment{ ObjectMeta: metav1.ObjectMeta{ Name: m.Name, Namespace: m.Namespace, }, Spec: appsv1.DeploymentSpec{ Replicas: &replicas, Selector: &metav1.LabelSelector{ MatchLabels: ls, }, Template: corev1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: ls, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{{ Image: "memcached:1.4.36-alpine", Name: "memcached", Command: []string{"memcached", "-m=64", "-o", "modern", "-v"}, Ports: []corev1.ContainerPort{{ ContainerPort: 11211, Name: "memcached", }}, }}, }, }, }, } // Set Memcached instance as the owner and controller ctrl.SetControllerReference(m, dep, r.Scheme) return dep } // labelsForMemcached returns the labels for selecting the resources // belonging to the given memcached CR name. func labelsForMemcached(name string) map[string]string { return map[string]string{"app": "memcached", "memcached_cr": name} } // getPodNames returns the pod names of the array of pods passed in func getPodNames(pods []corev1.Pod) []string { var podNames []string for _, pod := range pods { podNames = append(podNames, pod.Name) } return podNames } // SetupWithManager sets up the controller with the Manager. func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&cachev1.Memcached{}). Owns(&appsv1.Deployment{}). Complete(r) }
import ( appsv1 "k8s.io/api/apps/v1" ) func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&cachev1.Memcached{}). Owns(&appsv1.Deployment{}). Complete(r) }
func (r *MemcachedReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&cachev1.Memcached{}). Owns(&appsv1.Deployment{}). WithOptions(controller.Options{ MaxConcurrentReconciles: 2, }). Complete(r) }
import ( ctrl "sigs.k8s.io/controller-runtime" cachev1 "github.com/example-inc/memcached-operator/api/v1" ) func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { // Lookup the Memcached instance for this reconcile request memcached := &cachev1.Memcached{} err := r.Get(ctx, req.NamespacedName, memcached) }
// Reconcile successful - don't requeue return ctrl.Result{}, nil // Reconcile failed due to error - requeue return ctrl.Result{}, err // Requeue for any reason other than an error return ctrl.Result{Requeue: true}, nil
import "time" // Reconcile for any reason other than an error after 5 seconds return ctrl.Result{RequeueAfter: time.Second*5}, nil
// +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch // +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update // +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list; func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { }
import ( "github.com/operator-framework/operator-lib/proxy" )
for i, container := range dep.Spec.Template.Spec.Containers { dep.Spec.Template.Spec.Containers[i].Env = append(container.Env, proxy.ReadProxyVarsFromEnv()...) }
containers: - args: - --leader-elect - --leader-election-id=ansible-proxy-demo image: controller:latest name: manager env: - name: "HTTP_PROXY" value: "http_proxy_test"
make install run
2021-01-10T21:09:29.016-0700 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"} 2021-01-10T21:09:29.017-0700 INFO setup starting manager 2021-01-10T21:09:29.017-0700 INFO controller-runtime.manager starting metrics server {"path": "/metrics"} 2021-01-10T21:09:29.018-0700 INFO controller-runtime.manager.controller.memcached Starting EventSource {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "source": "kind source: /, Kind="} 2021-01-10T21:09:29.218-0700 INFO controller-runtime.manager.controller.memcached Starting Controller {"reconciler group": "cache.example.com", "reconciler kind": "Memcached"} 2021-01-10T21:09:29.218-0700 INFO controller-runtime.manager.controller.memcached Starting workers {"reconciler group": "cache.example.com", "reconciler kind": "Memcached", "worker count": 1}
make docker-build IMG=<registry>/<user>/<image_name>:<tag>
make docker-push IMG=<registry>/<user>/<image_name>:<tag>
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc get deployment -n <project_name>-system
NAME READY UP-TO-DATE AVAILABLE AGE <project_name>-controller-manager 1/1 1 1 8m
make docker-build IMG=<registry>/<user>/<operator_image_name>:<tag>
make docker-push IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle-build BUNDLE_IMG=<registry>/<user>/<bundle_image_name>:<tag>
docker push <registry>/<user>/<bundle_image_name>:<tag>
operator-sdk run bundle [-n <namespace>] \ 1 <registry>/<user>/<bundle_image_name>:<tag>
oc project memcached-operator-system
apiVersion: cache.example.com/v1 kind: Memcached metadata: name: memcached-sample spec: size: 3
oc apply -f config/samples/cache_v1_memcached.yaml
oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE memcached-operator-controller-manager 1/1 1 1 8m memcached-sample 3/3 3 3 1m
oc get pods
NAME READY STATUS RESTARTS AGE memcached-sample-6fd7c98d8-7dqdr 1/1 Running 0 1m memcached-sample-6fd7c98d8-g5k7v 1/1 Running 0 1m memcached-sample-6fd7c98d8-m7vn7 1/1 Running 0 1m
oc get memcached/memcached-sample -o yaml
apiVersion: cache.example.com/v1 kind: Memcached metadata: name: memcached-sample spec: size: 3 status: nodes: - memcached-sample-6fd7c98d8-7dqdr - memcached-sample-6fd7c98d8-g5k7v - memcached-sample-6fd7c98d8-m7vn7
oc patch memcached memcached-sample -p '{"spec":{"size": 5}}' --type=merge
oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE memcached-operator-controller-manager 1/1 1 1 10m memcached-sample 5/5 5 5 3m
oc delete -f config/samples/cache_v1_memcached.yaml
make undeploy
operator-sdk cleanup <project_name>
mkdir memcached-operator
cd memcached-operator
operator-sdk init --plugins=ansible --domain=example.com
operator-sdk create api --group cache --version v1 --kind Memcached --generate-role 1
make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>
make install
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc apply -f config/samples/cache_v1_memcached.yaml -n memcached-operator-system
oc logs deployment.apps/memcached-operator-controller-manager -c manager -n memcached-operator-system
I0205 17:48:45.881666 7 leaderelection.go:253] successfully acquired lease memcached-operator-system/memcached-operator {"level":"info","ts":1612547325.8819902,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting EventSource","source":"kind source: cache.example.com/v1, Kind=Memcached"} {"level":"info","ts":1612547325.98242,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting Controller"} {"level":"info","ts":1612547325.9824686,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting workers","worker count":4} {"level":"info","ts":1612547348.8311093,"logger":"runner","msg":"Ansible-runner exited successfully","job":"4037200794235010051","name":"memcached-sample","namespace":"memcached-operator-system"}
oc delete -f config/samples/cache_v1_memcached -n memcached-operator-system
make undeploy
mkdir -p USDHOME/projects/memcached-operator
cd USDHOME/projects/memcached-operator
operator-sdk init --plugins=ansible --domain=example.com
domain: example.com layout: - ansible.sdk.operatorframework.io/v1 plugins: manifests.sdk.operatorframework.io/v2: {} scorecard.sdk.operatorframework.io/v2: {} sdk.x-openshift.io/v1: {} projectName: memcached-operator version: "3"
operator-sdk create api --group cache --version v1 --kind Memcached --generate-role 1
--- - name: start memcached k8s: definition: kind: Deployment apiVersion: apps/v1 metadata: name: '{{ ansible_operator_meta.name }}-memcached' namespace: '{{ ansible_operator_meta.namespace }}' spec: replicas: "{{size}}" selector: matchLabels: app: memcached template: metadata: labels: app: memcached spec: containers: - name: memcached command: - memcached - -m=64 - -o - modern - -v image: "docker.io/memcached:1.4.36-alpine" ports: - containerPort: 11211
--- defaults file for Memcached size: 1
apiVersion: cache.example.com/v1 kind: Memcached metadata: labels: app.kubernetes.io/name: memcached app.kubernetes.io/instance: memcached-sample app.kubernetes.io/part-of: memcached-operator app.kubernetes.io/managed-by: kustomize app.kubernetes.io/created-by: memcached-operator name: memcached-sample spec: size: 3
env: - name: HTTP_PROXY value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}' - name: http_proxy value: '{{ lookup("env", "HTTP_PROXY") | default("", True) }}'
containers: - args: - --leader-elect - --leader-election-id=ansible-proxy-demo image: controller:latest name: manager env: - name: "HTTP_PROXY" value: "http_proxy_test"
make install run
{"level":"info","ts":1612589622.7888272,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"} {"level":"info","ts":1612589622.7897573,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} {"level":"info","ts":1612589622.789971,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1612589622.7899997,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting EventSource","source":"kind source: cache.example.com/v1, Kind=Memcached"} {"level":"info","ts":1612589622.8904517,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting Controller"} {"level":"info","ts":1612589622.8905244,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting workers","worker count":8}
make docker-build IMG=<registry>/<user>/<image_name>:<tag>
make docker-push IMG=<registry>/<user>/<image_name>:<tag>
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc get deployment -n <project_name>-system
NAME READY UP-TO-DATE AVAILABLE AGE <project_name>-controller-manager 1/1 1 1 8m
make docker-build IMG=<registry>/<user>/<operator_image_name>:<tag>
make docker-push IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle-build BUNDLE_IMG=<registry>/<user>/<bundle_image_name>:<tag>
docker push <registry>/<user>/<bundle_image_name>:<tag>
operator-sdk run bundle [-n <namespace>] \ 1 <registry>/<user>/<bundle_image_name>:<tag>
oc project memcached-operator-system
apiVersion: cache.example.com/v1 kind: Memcached metadata: name: memcached-sample spec: size: 3
oc apply -f config/samples/cache_v1_memcached.yaml
oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE memcached-operator-controller-manager 1/1 1 1 8m memcached-sample 3/3 3 3 1m
oc get pods
NAME READY STATUS RESTARTS AGE memcached-sample-6fd7c98d8-7dqdr 1/1 Running 0 1m memcached-sample-6fd7c98d8-g5k7v 1/1 Running 0 1m memcached-sample-6fd7c98d8-m7vn7 1/1 Running 0 1m
oc get memcached/memcached-sample -o yaml
apiVersion: cache.example.com/v1 kind: Memcached metadata: name: memcached-sample spec: size: 3 status: nodes: - memcached-sample-6fd7c98d8-7dqdr - memcached-sample-6fd7c98d8-g5k7v - memcached-sample-6fd7c98d8-m7vn7
oc patch memcached memcached-sample -p '{"spec":{"size": 5}}' --type=merge
oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE memcached-operator-controller-manager 1/1 1 1 10m memcached-sample 5/5 5 5 3m
oc delete -f config/samples/cache_v1_memcached.yaml
make undeploy
operator-sdk cleanup <project_name>
apiVersion: "test1.example.com/v1alpha1" kind: "Test1" metadata: name: "example" annotations: ansible.operator-sdk/reconcile-period: "30s"
- version: v1alpha1 1 group: test1.example.com kind: Test1 role: /opt/ansible/roles/Test1 - version: v1alpha1 2 group: test2.example.com kind: Test2 playbook: /opt/ansible/playbook.yml - version: v1alpha1 3 group: test3.example.com kind: Test3 playbook: /opt/ansible/test3.yml reconcilePeriod: 0 manageStatus: false
- version: v1alpha1 group: app.example.com kind: AppService playbook: /opt/ansible/playbook.yml maxRunnerArtifacts: 30 reconcilePeriod: 5s manageStatus: False watchDependentResources: False
apiVersion: "app.example.com/v1alpha1" kind: "Database" metadata: name: "example" spec: message: "Hello world 2" newParameter: "newParam"
{ "meta": { "name": "<cr_name>", "namespace": "<cr_namespace>", }, "message": "Hello world 2", "new_parameter": "newParam", "_app_example_com_database": { <full_crd> }, }
--- - debug: msg: "name: {{ ansible_operator_meta.name }}, {{ ansible_operator_meta.namespace }}"
sudo dnf install ansible
pip3 install openshift
ansible-galaxy collection install community.kubernetes
ansible-galaxy collection install -r requirements.yml
--- - name: set ConfigMap example-config to {{ state }} community.kubernetes.k8s: api_version: v1 kind: ConfigMap name: example-config namespace: default 1 state: "{{ state }}" ignore_errors: true 2
--- state: present
--- - hosts: localhost roles: - <kind>
ansible-playbook playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [localhost] ******************************************************************************** TASK [Gathering Facts] ******************************************************************************** ok: [localhost] TASK [memcached : set ConfigMap example-config to present] ******************************************************************************** changed: [localhost] PLAY RECAP ******************************************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
oc get configmaps
NAME DATA AGE example-config 0 2m1s
ansible-playbook playbook.yml --extra-vars state=absent
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [localhost] ******************************************************************************** TASK [Gathering Facts] ******************************************************************************** ok: [localhost] TASK [memcached : set ConfigMap example-config to absent] ******************************************************************************** changed: [localhost] PLAY RECAP ******************************************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
oc get configmaps
apiVersion: "test1.example.com/v1alpha1" kind: "Test1" metadata: name: "example" annotations: ansible.operator-sdk/reconcile-period: "30s"
make install
/usr/bin/kustomize build config/crd | kubectl apply -f - customresourcedefinition.apiextensions.k8s.io/memcacheds.cache.example.com created
make run
/home/user/memcached-operator/bin/ansible-operator run {"level":"info","ts":1612739145.2871568,"logger":"cmd","msg":"Version","Go Version":"go1.15.5","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.10.1","commit":"1abf57985b43bf6a59dcd18147b3c574fa57d3f6"} {"level":"info","ts":1612739148.347306,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1612739148.3488882,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2} {"level":"info","ts":1612739148.3490262,"logger":"cmd","msg":"Environment variable not set; using default value","Namespace":"","envVar":"ANSIBLE_DEBUG_LOGS","ANSIBLE_DEBUG_LOGS":false} {"level":"info","ts":1612739148.3490646,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"} {"level":"info","ts":1612739148.350217,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} {"level":"info","ts":1612739148.3506632,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1612739148.350784,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting EventSource","source":"kind source: cache.example.com/v1, Kind=Memcached"} {"level":"info","ts":1612739148.5511978,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting Controller"} {"level":"info","ts":1612739148.5512562,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting workers","worker count":8}
apiVersion: <group>.example.com/v1alpha1 kind: <kind> metadata: name: "<kind>-sample"
oc apply -f config/samples/<gvk>.yaml
oc get configmaps
NAME STATUS AGE example-config Active 3s
apiVersion: cache.example.com/v1 kind: Memcached metadata: name: memcached-sample spec: state: absent
oc apply -f config/samples/<gvk>.yaml
oc get configmap
make docker-build IMG=<registry>/<user>/<image_name>:<tag>
make docker-push IMG=<registry>/<user>/<image_name>:<tag>
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc get deployment -n <project_name>-system
NAME READY UP-TO-DATE AVAILABLE AGE <project_name>-controller-manager 1/1 1 1 8m
oc logs deployment/<project_name>-controller-manager -c manager \ 1 -n <namespace> 2
{"level":"info","ts":1612732105.0579333,"logger":"cmd","msg":"Version","Go Version":"go1.15.5","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.10.1","commit":"1abf57985b43bf6a59dcd18147b3c574fa57d3f6"} {"level":"info","ts":1612732105.0587437,"logger":"cmd","msg":"WATCH_NAMESPACE environment variable not set. Watching all namespaces.","Namespace":""} I0207 21:08:26.110949 7 request.go:645] Throttling request took 1.035521578s, request: GET:https://172.30.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1alpha1?timeout=32s {"level":"info","ts":1612732107.768025,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"127.0.0.1:8080"} {"level":"info","ts":1612732107.768796,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2} {"level":"info","ts":1612732107.7688773,"logger":"cmd","msg":"Environment variable not set; using default value","Namespace":"","envVar":"ANSIBLE_DEBUG_LOGS","ANSIBLE_DEBUG_LOGS":false} {"level":"info","ts":1612732107.7688901,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"} {"level":"info","ts":1612732107.770032,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} I0207 21:08:27.770185 7 leaderelection.go:243] attempting to acquire leader lease memcached-operator-system/memcached-operator {"level":"info","ts":1612732107.770202,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} I0207 21:08:27.784854 7 leaderelection.go:253] successfully acquired lease memcached-operator-system/memcached-operator {"level":"info","ts":1612732107.7850506,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting EventSource","source":"kind source: cache.example.com/v1, Kind=Memcached"} {"level":"info","ts":1612732107.8853772,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting Controller"} {"level":"info","ts":1612732107.8854098,"logger":"controller-runtime.manager.controller.memcached-controller","msg":"Starting workers","worker count":4}
containers: - name: manager env: - name: ANSIBLE_DEBUG_LOGS value: "True"
apiVersion: "cache.example.com/v1alpha1" kind: "Memcached" metadata: name: "example-memcached" annotations: "ansible.sdk.operatorframework.io/verbosity": "4" spec: size: 4
status: conditions: - ansibleResult: changed: 3 completion: 2018-12-03T13:45:57.13329 failures: 1 ok: 6 skipped: 0 lastTransitionTime: 2018-12-03T13:45:57Z message: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 113] No route to host>' reason: Failed status: "True" type: Failure - lastTransitionTime: 2018-12-03T13:46:13Z message: Running reconciliation reason: Running status: "True" type: Running
- version: v1 group: api.example.com kind: <kind> role: <role> manageStatus: false
- operator_sdk.util.k8s_status: api_version: app.example.com/v1 kind: <kind> name: "{{ ansible_operator_meta.name }}" namespace: "{{ ansible_operator_meta.namespace }}" status: test: data
collections: - operator_sdk.util
k8s_status: status: key1: value1
mkdir nginx-operator
cd nginx-operator
operator-sdk init --plugins=helm
operator-sdk create api --group demo --version v1 --kind Nginx
make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>
make install
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc adm policy add-scc-to-user anyuid system:serviceaccount:nginx-operator-system:nginx-sample
oc apply -f config/samples/demo_v1_nginx.yaml -n nginx-operator-system
oc logs deployment.apps/nginx-operator-controller-manager -c manager -n nginx-operator-system
oc delete -f config/samples/demo_v1_nginx -n nginx-operator-system
make undeploy
mkdir -p USDHOME/projects/nginx-operator
cd USDHOME/projects/nginx-operator
operator-sdk init --plugins=helm --domain=example.com --group=demo --version=v1 --kind=Nginx
operator-sdk init --plugins helm --help
domain: example.com layout: - helm.sdk.operatorframework.io/v1 plugins: manifests.sdk.operatorframework.io/v2: {} scorecard.sdk.operatorframework.io/v2: {} sdk.x-openshift.io/v1: {} projectName: nginx-operator resources: - api: crdVersion: v1 namespaced: true domain: example.com group: demo kind: Nginx version: v1 version: "3"
Use the 'create api' subcommand to add watches to this file. - group: demo version: v1 kind: Nginx chart: helm-charts/nginx +kubebuilder:scaffold:watch
apiVersion: demo.example.com/v1 kind: Nginx metadata: name: nginx-sample spec: replicaCount: 2
apiVersion: demo.example.com/v1 kind: Nginx metadata: name: nginx-sample spec: replicaCount: 2 service: port: 8080
- group: demo.example.com version: v1alpha1 kind: Nginx chart: helm-charts/nginx overrideValues: proxy.http: USDHTTP_PROXY
proxy: http: "" https: "" no_proxy: ""
containers: - name: {{ .Chart.Name }} securityContext: - toYaml {{ .Values.securityContext | nindent 12 }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.image.pullPolicy }} env: - name: http_proxy value: "{{ .Values.proxy.http }}"
containers: - args: - --leader-elect - --leader-election-id=ansible-proxy-demo image: controller:latest name: manager env: - name: "HTTP_PROXY" value: "http_proxy_test"
make install run
{"level":"info","ts":1612652419.9289865,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1612652419.9296563,"logger":"helm.controller","msg":"Watching resource","apiVersion":"demo.example.com/v1","kind":"Nginx","namespace":"","reconcilePeriod":"1m0s"} {"level":"info","ts":1612652419.929983,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1612652419.930015,"logger":"controller-runtime.manager.controller.nginx-controller","msg":"Starting EventSource","source":"kind source: demo.example.com/v1, Kind=Nginx"} {"level":"info","ts":1612652420.2307851,"logger":"controller-runtime.manager.controller.nginx-controller","msg":"Starting Controller"} {"level":"info","ts":1612652420.2309358,"logger":"controller-runtime.manager.controller.nginx-controller","msg":"Starting workers","worker count":8}
make docker-build IMG=<registry>/<user>/<image_name>:<tag>
make docker-push IMG=<registry>/<user>/<image_name>:<tag>
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc get deployment -n <project_name>-system
NAME READY UP-TO-DATE AVAILABLE AGE <project_name>-controller-manager 1/1 1 1 8m
make docker-build IMG=<registry>/<user>/<operator_image_name>:<tag>
make docker-push IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle-build BUNDLE_IMG=<registry>/<user>/<bundle_image_name>:<tag>
docker push <registry>/<user>/<bundle_image_name>:<tag>
operator-sdk run bundle [-n <namespace>] \ 1 <registry>/<user>/<bundle_image_name>:<tag>
oc project nginx-operator-system
apiVersion: demo.example.com/v1 kind: Nginx metadata: name: nginx-sample spec: replicaCount: 3
oc adm policy add-scc-to-user anyuid system:serviceaccount:nginx-operator-system:nginx-sample
oc apply -f config/samples/demo_v1_nginx.yaml
oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE nginx-operator-controller-manager 1/1 1 1 8m nginx-sample 3/3 3 3 1m
oc get pods
NAME READY STATUS RESTARTS AGE nginx-sample-6fd7c98d8-7dqdr 1/1 Running 0 1m nginx-sample-6fd7c98d8-g5k7v 1/1 Running 0 1m nginx-sample-6fd7c98d8-m7vn7 1/1 Running 0 1m
oc get nginx/nginx-sample -o yaml
apiVersion: demo.example.com/v1 kind: Nginx metadata: name: nginx-sample spec: replicaCount: 3 status: nodes: - nginx-sample-6fd7c98d8-7dqdr - nginx-sample-6fd7c98d8-g5k7v - nginx-sample-6fd7c98d8-m7vn7
oc patch nginx nginx-sample -p '{"spec":{"replicaCount": 5}}' --type=merge
oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE nginx-operator-controller-manager 1/1 1 1 10m nginx-sample 5/5 5 5 3m
oc delete -f config/samples/demo_v1_nginx.yaml
make undeploy
operator-sdk cleanup <project_name>
apiVersion: apache.org/v1alpha1 kind: Tomcat metadata: name: example-app spec: replicaCount: 2
{{ .Values.replicaCount }}
oc get Tomcats --all-namespaces
mkdir -p USDHOME/github.com/example/memcached-operator
cd USDHOME/github.com/example/memcached-operator
operator-sdk init --plugins=hybrid.helm.sdk.operatorframework.io --project-version="3" --domain example.com --repo=github.com/example/memcached-operator
operator-sdk create api --plugins helm.sdk.operatorframework.io/v1 --group cache --version v1 --kind Memcached
operator-sdk create api --plugins helm.sdk.operatorframework.io/v1 --help
Use the 'create api' subcommand to add watches to this file. - group: cache.my.domain version: v1 kind: Memcached chart: helm-charts/memcached #+kubebuilder:scaffold:watch
// Operator's main.go // With the help of helpers provided in the library, the reconciler can be // configured here before starting the controller with this reconciler. reconciler := reconciler.New( reconciler.WithChart(*chart), reconciler.WithGroupVersionKind(gvk), ) if err := reconciler.SetupWithManager(mgr); err != nil { panic(fmt.Sprintf("unable to create reconciler: %s", err)) }
operator-sdk create api --group=cache --version v1 --kind MemcachedBackup --resource --controller --plugins=go/v3
Create Resource [y/n] y Create Controller [y/n] y
// MemcachedBackupSpec defines the desired state of MemcachedBackup type MemcachedBackupSpec struct { // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster // Important: Run "make" to regenerate code after modifying this file //+kubebuilder:validation:Minimum=0 // Size is the size of the memcached deployment Size int32 `json:"size"` } // MemcachedBackupStatus defines the observed state of MemcachedBackup type MemcachedBackupStatus struct { // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster // Important: Run "make" to regenerate code after modifying this file // Nodes are the names of the memcached pods Nodes []string `json:"nodes"` }
make generate
make manifests
for _, w := range ws { // Register controller with the factory reconcilePeriod := defaultReconcilePeriod if w.ReconcilePeriod != nil { reconcilePeriod = w.ReconcilePeriod.Duration } maxConcurrentReconciles := defaultMaxConcurrentReconciles if w.MaxConcurrentReconciles != nil { maxConcurrentReconciles = *w.MaxConcurrentReconciles } r, err := reconciler.New( reconciler.WithChart(*w.Chart), reconciler.WithGroupVersionKind(w.GroupVersionKind), reconciler.WithOverrideValues(w.OverrideValues), reconciler.SkipDependentWatches(w.WatchDependentResources != nil && !*w.WatchDependentResources), reconciler.WithMaxConcurrentReconciles(maxConcurrentReconciles), reconciler.WithReconcilePeriod(reconcilePeriod), reconciler.WithInstallAnnotations(annotation.DefaultInstallAnnotations...), reconciler.WithUpgradeAnnotations(annotation.DefaultUpgradeAnnotations...), reconciler.WithUninstallAnnotations(annotation.DefaultUninstallAnnotations...), )
// Setup manager with Go API if err = (&controllers.MemcachedBackupReconciler{ Client: mgr.GetClient(), Scheme: mgr.GetScheme(), }).SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "MemcachedBackup") os.Exit(1) } // Setup manager with Helm API for _, w := range ws { if err := r.SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "Helm") os.Exit(1) } setupLog.Info("configured watch", "gvk", w.GroupVersionKind, "chartPath", w.ChartPath, "maxConcurrentReconciles", maxConcurrentReconciles, "reconcilePeriod", reconcilePeriod) } // Start the manager if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil { setupLog.Error(err, "problem running manager") os.Exit(1) }
--- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: manager-role rules: - apiGroups: - "" resources: - namespaces verbs: - get - apiGroups: - apps resources: - deployments - daemonsets - replicasets - statefulsets verbs: - create - delete - get - list - patch - update - watch - apiGroups: - cache.my.domain resources: - memcachedbackups verbs: - create - delete - get - list - patch - update - watch - apiGroups: - cache.my.domain resources: - memcachedbackups/finalizers verbs: - create - delete - get - list - patch - update - watch - apiGroups: - "" resources: - pods - services - services/finalizers - endpoints - persistentvolumeclaims - events - configmaps - secrets - serviceaccounts verbs: - create - delete - get - list - patch - update - watch - apiGroups: - cache.my.domain resources: - memcachedbackups/status verbs: - get - patch - update - apiGroups: - policy resources: - events - poddisruptionbudgets verbs: - create - delete - get - list - patch - update - watch - apiGroups: - cache.my.domain resources: - memcacheds - memcacheds/status - memcacheds/finalizers verbs: - create - delete - get - list - patch - update - watch
make install run
make docker-build IMG=<registry>/<user>/<image_name>:<tag>
make docker-push IMG=<registry>/<user>/<image_name>:<tag>
make deploy IMG=<registry>/<user>/<image_name>:<tag>
oc get deployment -n <project_name>-system
NAME READY UP-TO-DATE AVAILABLE AGE <project_name>-controller-manager 1/1 1 1 8m
oc project <project_name>-system
apiVersion: cache.my.domain/v1 kind: Memcached metadata: name: memcached-sample spec: # Default values copied from <project_dir>/helm-charts/memcached/values.yaml affinity: {} autoscaling: enabled: false maxReplicas: 100 minReplicas: 1 targetCPUUtilizationPercentage: 80 fullnameOverride: "" image: pullPolicy: IfNotPresent repository: nginx tag: "" imagePullSecrets: [] ingress: annotations: {} className: "" enabled: false hosts: - host: chart-example.local paths: - path: / pathType: ImplementationSpecific tls: [] nameOverride: "" nodeSelector: {} podAnnotations: {} podSecurityContext: {} replicaCount: 3 resources: {} securityContext: {} service: port: 80 type: ClusterIP serviceAccount: annotations: {} create: true name: "" tolerations: []
oc apply -f config/samples/cache_v1_memcached.yaml
oc get pods
NAME READY STATUS RESTARTS AGE memcached-sample-6fd7c98d8-7dqdr 1/1 Running 0 18m memcached-sample-6fd7c98d8-g5k7v 1/1 Running 0 18m memcached-sample-6fd7c98d8-m7vn7 1/1 Running 0 18m
apiVersion: cache.my.domain/v1 kind: MemcachedBackup metadata: name: memcachedbackup-sample spec: size: 2
oc apply -f config/samples/cache_v1_memcachedbackup.yaml
oc get pods
NAME READY STATUS RESTARTS AGE memcachedbackup-sample-8649699989-4bbzg 1/1 Running 0 22m memcachedbackup-sample-8649699989-mq6mx 1/1 Running 0 22m
oc delete -f config/samples/cache_v1_memcached.yaml
oc delete -f config/samples/cache_v1_memcachedbackup.yaml
make undeploy
operators.openshift.io/infrastructure-features: '["disconnected", "proxy-aware"]'
operators.openshift.io/valid-subscription: '["OpenShift Container Platform"]'
operators.openshift.io/valid-subscription: '["3Scale Commercial License", "Red Hat Managed Integration"]'
operators.openshift.io/infrastructure-features: '["disconnected", "proxy-aware"]' operators.openshift.io/valid-subscription: '["OpenShift Container Platform"]'
spec: spec: containers: - command: - /manager env: - name: <related_image_environment_variable> 1 value: "<related_image_reference_with_tag>" 2
// deploymentForMemcached returns a memcached Deployment object Spec: corev1.PodSpec{ Containers: []corev1.Container{{ - Image: "memcached:1.4.36-alpine", 1 + Image: os.Getenv("<related_image_environment_variable>"), 2 Name: "memcached", Command: []string{"memcached", "-m=64", "-o", "modern", "-v"}, Ports: []corev1.ContainerPort{{
spec: containers: - name: memcached command: - memcached - -m=64 - -o - modern - -v - image: "docker.io/memcached:1.4.36-alpine" 1 + image: "{{ lookup('env', '<related_image_environment_variable>') }}" 2 ports: - containerPort: 11211
- group: demo.example.com version: v1alpha1 kind: Memcached chart: helm-charts/memcached overrideValues: 1 relatedImage: USD{<related_image_environment_variable>} 2
relatedImage: ""
containers: - name: {{ .Chart.Name }} securityContext: - toYaml {{ .Values.securityContext | nindent 12 }} image: "{{ .Values.image.pullPolicy }} env: 1 - name: related_image 2 value: "{{ .Values.relatedImage }}" 3
BUNDLE_GEN_FLAGS ?= -q --overwrite --version USD(VERSION) USD(BUNDLE_METADATA_OPTS) # USE_IMAGE_DIGESTS defines if images are resolved via tags or digests # You can enable this value if you would like to use SHA Based Digests # To enable set flag to true USE_IMAGE_DIGESTS ?= false ifeq (USD(USE_IMAGE_DIGESTS), true) BUNDLE_GEN_FLAGS += --use-image-digests endif - USD(KUSTOMIZE) build config/manifests | operator-sdk generate bundle -q --overwrite --version USD(VERSION) USD(BUNDLE_METADATA_OPTS) 1 + USD(KUSTOMIZE) build config/manifests | operator-sdk generate bundle USD(BUNDLE_GEN_FLAGS) 2
make bundle USE_IMAGE_DIGESTS=true
metadata: annotations: operators.openshift.io/infrastructure-features: '["disconnected"]'
labels: operatorframework.io/arch.<arch>: supported 1 operatorframework.io/os.<os>: supported 2
labels: operatorframework.io/os.linux: supported
labels: operatorframework.io/arch.amd64: supported
labels: operatorframework.io/arch.s390x: supported operatorframework.io/os.zos: supported operatorframework.io/os.linux: supported 1 operatorframework.io/arch.amd64: supported 2
metadata: annotations: operatorframework.io/suggested-namespace: <namespace> 1
module github.com/example-inc/memcached-operator go 1.15 require ( k8s.io/apimachinery v0.19.2 k8s.io/client-go v0.19.2 sigs.k8s.io/controller-runtime v0.7.0 operator-framework/operator-lib v0.3.0 )
import ( apiv1 "github.com/operator-framework/api/pkg/operators/v1" ) func NewUpgradeable(cl client.Client) (Condition, error) { return NewCondition(cl, "apiv1.OperatorUpgradeable") } cond, err := NewUpgradeable(cl);
apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: name: webhook-operator.v0.0.1 spec: customresourcedefinitions: owned: - kind: WebhookTest name: webhooktests.webhook.operators.coreos.io 1 version: v1 install: spec: deployments: - name: webhook-operator-webhook strategy: deployment installModes: - supported: false type: OwnNamespace - supported: false type: SingleNamespace - supported: false type: MultiNamespace - supported: true type: AllNamespaces webhookdefinitions: - type: ValidatingAdmissionWebhook 2 admissionReviewVersions: - v1beta1 - v1 containerPort: 443 targetPort: 4343 deploymentName: webhook-operator-webhook failurePolicy: Fail generateName: vwebhooktest.kb.io rules: - apiGroups: - webhook.operators.coreos.io apiVersions: - v1 operations: - CREATE - UPDATE resources: - webhooktests sideEffects: None webhookPath: /validate-webhook-operators-coreos-io-v1-webhooktest - type: MutatingAdmissionWebhook 3 admissionReviewVersions: - v1beta1 - v1 containerPort: 443 targetPort: 4343 deploymentName: webhook-operator-webhook failurePolicy: Fail generateName: mwebhooktest.kb.io rules: - apiGroups: - webhook.operators.coreos.io apiVersions: - v1 operations: - CREATE - UPDATE resources: - webhooktests sideEffects: None webhookPath: /mutate-webhook-operators-coreos-io-v1-webhooktest - type: ConversionWebhook 4 admissionReviewVersions: - v1beta1 - v1 containerPort: 443 targetPort: 4343 deploymentName: webhook-operator-webhook generateName: cwebhooktest.kb.io sideEffects: None webhookPath: /convert conversionCRDs: - webhooktests.webhook.operators.coreos.io 5
- displayName: MongoDB Standalone group: mongodb.com kind: MongoDbStandalone name: mongodbstandalones.mongodb.com resources: - kind: Service name: '' version: v1 - kind: StatefulSet name: '' version: v1beta2 - kind: Pod name: '' version: v1 - kind: ConfigMap name: '' version: v1 specDescriptors: - description: Credentials for Ops Manager or Cloud Manager. displayName: Credentials path: credentials x-descriptors: - 'urn:alm:descriptor:com.tectonic.ui:selector:core:v1:Secret' - description: Project this deployment belongs to. displayName: Project path: project x-descriptors: - 'urn:alm:descriptor:com.tectonic.ui:selector:core:v1:ConfigMap' - description: MongoDB version to be installed. displayName: Version path: version x-descriptors: - 'urn:alm:descriptor:com.tectonic.ui:label' statusDescriptors: - description: The status of each of the pods for the MongoDB cluster. displayName: Pod Status path: pods x-descriptors: - 'urn:alm:descriptor:com.tectonic.ui:podStatuses' version: v1 description: >- MongoDB Deployment consisting of only one host. No replication of data.
required: - name: etcdclusters.etcd.database.coreos.com version: v1beta2 kind: EtcdCluster displayName: etcd Cluster description: Represents a cluster of etcd nodes.
versions: - name: v1alpha1 served: true storage: false - name: v1beta1 1 served: true storage: true
customresourcedefinitions: owned: - name: cluster.example.com version: v1beta1 1 kind: cluster displayName: Cluster
versions: - name: v1alpha1 served: false 1 storage: true
versions: - name: v1alpha1 served: false storage: false 1 - name: v1beta1 served: true storage: true 2
versions: - name: v1beta1 served: true storage: true
metadata: annotations: alm-examples: >- [{"apiVersion":"etcd.database.coreos.com/v1beta2","kind":"EtcdCluster","metadata":{"name":"example","namespace":"default"},"spec":{"size":3,"version":"3.2.13"}},{"apiVersion":"etcd.database.coreos.com/v1beta2","kind":"EtcdRestore","metadata":{"name":"example-etcd-cluster"},"spec":{"etcdCluster":{"name":"example-etcd-cluster"},"backupStorageType":"S3","s3":{"path":"<full-s3-path>","awsSecret":"<aws-secret>"}}},{"apiVersion":"etcd.database.coreos.com/v1beta2","kind":"EtcdBackup","metadata":{"name":"example-etcd-cluster-backup"},"spec":{"etcdEndpoints":["<etcd-cluster-endpoints>"],"storageType":"S3","s3":{"path":"<full-s3-path>","awsSecret":"<aws-secret>"}}}]
apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: name: my-operator-v1.2.3 annotations: operators.operatorframework.io/internal-objects: '["my.internal.crd1.io","my.internal.crd2.io"]' 1
apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: name: my-operator-v1.2.3 annotations: operatorframework.io/initialization-resource: |- { "apiVersion": "ocs.openshift.io/v1", "kind": "StorageCluster", "metadata": { "name": "example-storagecluster" }, "spec": { "manageNodes": false, "monPVCTemplate": { "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "10Gi" } }, "storageClassName": "gp2" } }, "storageDeviceSets": [ { "count": 3, "dataPVCTemplate": { "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "1Ti" } }, "storageClassName": "gp2", "volumeMode": "Block" } }, "name": "example-deviceset", "placement": {}, "portable": true, "resources": {} } ] } }
make docker-build IMG=<registry>/<user>/<operator_image_name>:<tag>
make docker-push IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle IMG=<registry>/<user>/<operator_image_name>:<tag>
make bundle-build BUNDLE_IMG=<registry>/<user>/<bundle_image_name>:<tag>
docker push <registry>/<user>/<bundle_image_name>:<tag>
operator-sdk run bundle [-n <namespace>] \ 1 <registry>/<user>/<bundle_image_name>:<tag>
make catalog-build CATALOG_IMG=<registry>/<user>/<index_image_name>:<tag>
make catalog-push CATALOG_IMG=<registry>/<user>/<index_image_name>:<tag>
make bundle-build bundle-push catalog-build catalog-push BUNDLE_IMG=<bundle_image_pull_spec> CATALOG_IMG=<index_image_pull_spec>
IMAGE_TAG_BASE=quay.io/example/my-operator
make bundle-build bundle-push catalog-build catalog-push
apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: cs-memcached namespace: default spec: displayName: My Test publisher: Company sourceType: grpc image: quay.io/example/memcached-catalog:v0.0.1 1 updateStrategy: registryPoll: interval: 10m
oc get catalogsource
NAME DISPLAY TYPE PUBLISHER AGE cs-memcached My Test grpc Company 4h31m
apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: my-test namespace: default spec: targetNamespaces: - default
\ufeffapiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: catalogtest namespace: default spec: channel: "alpha" installPlanApproval: Manual name: catalog source: cs-memcached sourceNamespace: default startingCSV: memcached-operator.v0.0.1
oc get og
NAME AGE my-test 4h40m
oc get csv
NAME DISPLAY VERSION REPLACES PHASE memcached-operator.v0.0.1 Test 0.0.1 Succeeded
oc get pods
NAME READY STATUS RESTARTS AGE 9098d908802769fbde8bd45255e69710a9f8420a8f3d814abe88b68f8ervdj6 0/1 Completed 0 4h33m catalog-controller-manager-7fd5b7b987-69s4n 2/2 Running 0 4h32m cs-memcached-7622r 1/1 Running 0 4h33m
operator-sdk run bundle <registry>/<user>/memcached-operator:v0.0.1
INFO[0009] Successfully created registry pod: quay-io-demo-memcached-operator-v0-0-1 INFO[0009] Created CatalogSource: memcached-operator-catalog INFO[0010] OperatorGroup "operator-sdk-og" created INFO[0010] Created Subscription: memcached-operator-v0-0-1-sub INFO[0013] Approved InstallPlan install-bqggr for the Subscription: memcached-operator-v0-0-1-sub INFO[0013] Waiting for ClusterServiceVersion "my-project/memcached-operator.v0.0.1" to reach 'Succeeded' phase INFO[0013] Waiting for ClusterServiceVersion "my-project/memcached-operator.v0.0.1" to appear INFO[0019] Found ClusterServiceVersion "my-project/memcached-operator.v0.0.1" phase: Succeeded
operator-sdk run bundle-upgrade <registry>/<user>/memcached-operator:v0.0.2
INFO[0002] Found existing subscription with name memcached-operator-v0-0-1-sub and namespace my-project INFO[0002] Found existing catalog source with name memcached-operator-catalog and namespace my-project INFO[0009] Successfully created registry pod: quay-io-demo-memcached-operator-v0-0-2 INFO[0009] Updated catalog source memcached-operator-catalog with address and annotations INFO[0010] Deleted previous registry pod with name "quay-io-demo-memcached-operator-v0-0-1" INFO[0041] Approved InstallPlan install-gvcjh for the Subscription: memcached-operator-v0-0-1-sub INFO[0042] Waiting for ClusterServiceVersion "my-project/memcached-operator.v0.0.2" to reach 'Succeeded' phase INFO[0042] Found ClusterServiceVersion "my-project/memcached-operator.v0.0.2" phase: InstallReady INFO[0043] Found ClusterServiceVersion "my-project/memcached-operator.v0.0.2" phase: Installing INFO[0044] Found ClusterServiceVersion "my-project/memcached-operator.v0.0.2" phase: Succeeded INFO[0044] Successfully upgraded to "memcached-operator.v0.0.2"
operator-sdk cleanup memcached-operator
apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: annotations: "olm.properties": '[{"type": "olm.maxOpenShiftVersion", "value": "<cluster_version>"}]' 1
com.redhat.openshift.versions: "v4.7-v4.9" 1
LABEL com.redhat.openshift.versions="<versions>" 1
operator-sdk scorecard <bundle_dir_or_image> [flags]
operator-sdk scorecard -h
./bundle └── tests └── scorecard └── config.yaml
kind: Configuration apiversion: scorecard.operatorframework.io/v1alpha3 metadata: name: config stages: - parallel: true tests: - image: quay.io/operator-framework/scorecard-test:v1.16.0 entrypoint: - scorecard-test - basic-check-spec labels: suite: basic test: basic-check-spec-test - image: quay.io/operator-framework/scorecard-test:v1.16.0 entrypoint: - scorecard-test - olm-bundle-validation labels: suite: olm test: olm-bundle-validation-test
make bundle
operator-sdk scorecard <bundle_dir_or_image>
{ "apiVersion": "scorecard.operatorframework.io/v1alpha3", "kind": "TestList", "items": [ { "kind": "Test", "apiVersion": "scorecard.operatorframework.io/v1alpha3", "spec": { "image": "quay.io/operator-framework/scorecard-test:v1.16.0", "entrypoint": [ "scorecard-test", "olm-bundle-validation" ], "labels": { "suite": "olm", "test": "olm-bundle-validation-test" } }, "status": { "results": [ { "name": "olm-bundle-validation", "log": "time=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Found annotations file\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Could not find optional dependencies file\" name=bundle-test\n", "state": "pass" } ] } } ] }
-------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.16.0 Entrypoint: [scorecard-test olm-bundle-validation] Labels: "suite":"olm" "test":"olm-bundle-validation-test" Results: Name: olm-bundle-validation State: pass Log: time="2020-07-15T03:19:02Z" level=debug msg="Found manifests directory" name=bundle-test time="2020-07-15T03:19:02Z" level=debug msg="Found metadata directory" name=bundle-test time="2020-07-15T03:19:02Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test time="2020-07-15T03:19:02Z" level=info msg="Found annotations file" name=bundle-test time="2020-07-15T03:19:02Z" level=info msg="Could not find optional dependencies file" name=bundle-test
operator-sdk scorecard <bundle_dir_or_image> -o text --selector=test=basic-check-spec-test
operator-sdk scorecard <bundle_dir_or_image> -o text --selector=suite=olm
operator-sdk scorecard <bundle_dir_or_image> -o text --selector='test in (basic-check-spec-test,olm-bundle-validation-test)'
apiVersion: scorecard.operatorframework.io/v1alpha3 kind: Configuration metadata: name: config stages: - parallel: true 1 tests: - entrypoint: - scorecard-test - basic-check-spec image: quay.io/operator-framework/scorecard-test:v1.16.0 labels: suite: basic test: basic-check-spec-test - entrypoint: - scorecard-test - olm-bundle-validation image: quay.io/operator-framework/scorecard-test:v1.16.0 labels: suite: olm test: olm-bundle-validation-test
// Copyright 2020 The Operator-SDK Authors // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package main import ( "encoding/json" "fmt" "log" "os" scapiv1alpha3 "github.com/operator-framework/api/pkg/apis/scorecard/v1alpha3" apimanifests "github.com/operator-framework/api/pkg/manifests" ) // This is the custom scorecard test example binary // As with the Redhat scorecard test image, the bundle that is under // test is expected to be mounted so that tests can inspect the // bundle contents as part of their test implementations. // The actual test is to be run is named and that name is passed // as an argument to this binary. This argument mechanism allows // this binary to run various tests all from within a single // test image. const PodBundleRoot = "/bundle" func main() { entrypoint := os.Args[1:] if len(entrypoint) == 0 { log.Fatal("Test name argument is required") } // Read the pod's untar'd bundle from a well-known path. cfg, err := apimanifests.GetBundleFromDir(PodBundleRoot) if err != nil { log.Fatal(err.Error()) } var result scapiv1alpha3.TestStatus // Names of the custom tests which would be passed in the // `operator-sdk` command. switch entrypoint[0] { case CustomTest1Name: result = CustomTest1(cfg) case CustomTest2Name: result = CustomTest2(cfg) default: result = printValidTests() } // Convert scapiv1alpha3.TestResult to json. prettyJSON, err := json.MarshalIndent(result, "", " ") if err != nil { log.Fatal("Failed to generate json", err) } fmt.Printf("%s\n", string(prettyJSON)) } // printValidTests will print out full list of test names to give a hint to the end user on what the valid tests are. func printValidTests() scapiv1alpha3.TestStatus { result := scapiv1alpha3.TestResult{} result.State = scapiv1alpha3.FailState result.Errors = make([]string, 0) result.Suggestions = make([]string, 0) str := fmt.Sprintf("Valid tests for this image include: %s %s", CustomTest1Name, CustomTest2Name) result.Errors = append(result.Errors, str) return scapiv1alpha3.TestStatus{ Results: []scapiv1alpha3.TestResult{result}, } } const ( CustomTest1Name = "customtest1" CustomTest2Name = "customtest2" ) // Define any operator specific custom tests here. // CustomTest1 and CustomTest2 are example test functions. Relevant operator specific // test logic is to be implemented in similarly. func CustomTest1(bundle *apimanifests.Bundle) scapiv1alpha3.TestStatus { r := scapiv1alpha3.TestResult{} r.Name = CustomTest1Name r.State = scapiv1alpha3.PassState r.Errors = make([]string, 0) r.Suggestions = make([]string, 0) almExamples := bundle.CSV.GetAnnotations()["alm-examples"] if almExamples == "" { fmt.Println("no alm-examples in the bundle CSV") } return wrapResult(r) } func CustomTest2(bundle *apimanifests.Bundle) scapiv1alpha3.TestStatus { r := scapiv1alpha3.TestResult{} r.Name = CustomTest2Name r.State = scapiv1alpha3.PassState r.Errors = make([]string, 0) r.Suggestions = make([]string, 0) almExamples := bundle.CSV.GetAnnotations()["alm-examples"] if almExamples == "" { fmt.Println("no alm-examples in the bundle CSV") } return wrapResult(r) } func wrapResult(r scapiv1alpha3.TestResult) scapiv1alpha3.TestStatus { return scapiv1alpha3.TestStatus{ Results: []scapiv1alpha3.TestResult{r}, } }
// Simple query nn := types.NamespacedName{ Name: "cluster", } infraConfig := &configv1.Infrastructure{} err = crClient.Get(context.Background(), nn, infraConfig) if err != nil { return err } fmt.Printf("using crclient: %v\n", infraConfig.Status.ControlPlaneTopology) fmt.Printf("using crclient: %v\n", infraConfig.Status.InfrastructureTopology)
operatorConfigInformer := configinformer.NewSharedInformerFactoryWithOptions(configClient, 2*time.Second) infrastructureLister = operatorConfigInformer.Config().V1().Infrastructures().Lister() infraConfig, err := configClient.ConfigV1().Infrastructures().Get(context.Background(), "cluster", metav1.GetOptions{}) if err != nil { return err } // fmt.Printf("%v\n", infraConfig) fmt.Printf("%v\n", infraConfig.Status.ControlPlaneTopology) fmt.Printf("%v\n", infraConfig.Status.InfrastructureTopology)
../prometheus
package controllers import ( "github.com/prometheus/client_golang/prometheus" "sigs.k8s.io/controller-runtime/pkg/metrics" ) var ( widgets = prometheus.NewCounter( prometheus.CounterOpts{ Name: "widgets_total", Help: "Number of widgets processed", }, ) widgetFailures = prometheus.NewCounter( prometheus.CounterOpts{ Name: "widget_failures_total", Help: "Number of failed widgets", }, ) ) func init() { // Register custom metrics with the global prometheus registry metrics.Registry.MustRegister(widgets, widgetFailures) }
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { // Add metrics widgets.Inc() widgetFailures.Inc() return ctrl.Result{}, nil }
make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>
make deploy IMG=<registry>/<user>/<image_name>:<tag>
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus-k8s-role namespace: <operator_namespace> rules: - apiGroups: - "" resources: - endpoints - pods - services - nodes - secrets verbs: - get - list - watch
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus-k8s-rolebinding namespace: memcached-operator-system roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus-k8s-role subjects: - kind: ServiceAccount name: prometheus-k8s namespace: openshift-monitoring
oc apply -f config/prometheus/role.yaml
oc apply -f config/prometheus/rolebinding.yaml
oc label namespace <operator_namespace> openshift.io/cluster-monitoring="true"
operator-sdk init --plugins=ansible --domain=testmetrics.com
operator-sdk create api --group metrics --version v1 --kind Testmetrics --generate-role
--- tasks file for Memcached - name: start k8sstatus k8s: definition: kind: Deployment apiVersion: apps/v1 metadata: name: '{{ ansible_operator_meta.name }}-memcached' namespace: '{{ ansible_operator_meta.namespace }}' spec: replicas: "{{size}}" selector: matchLabels: app: memcached template: metadata: labels: app: memcached spec: containers: - name: memcached command: - memcached - -m=64 - -o - modern - -v image: "docker.io/memcached:1.4.36-alpine" ports: - containerPort: 11211 - osdk_metric: name: my_thing_counter description: This metric counts things counter: {} - osdk_metric: name: my_counter_metric description: Add 3.14 to the counter counter: increment: yes - osdk_metric: name: my_gauge_metric description: Create my gauge and set it to 2. gauge: set: 2 - osdk_metric: name: my_histogram_metric description: Observe my histogram histogram: observe: 2 - osdk_metric: name: my_summary_metric description: Observe my summary summary: observe: 2
make docker-build docker-push IMG=<registry>/<user>/<image_name>:<tag>
make install
make deploy IMG=<registry>/<user>/<image_name>:<tag>
apiVersion: metrics.testmetrics.com/v1 kind: Testmetrics metadata: name: testmetrics-sample spec: size: 1
oc create -f config/samples/metrics_v1_testmetrics.yaml
oc get pods
NAME READY STATUS RESTARTS AGE ansiblemetrics-controller-manager-<id> 2/2 Running 0 149m testmetrics-sample-memcached-<id> 1/1 Running 0 147m
oc get ep
NAME ENDPOINTS AGE ansiblemetrics-controller-manager-metrics-service 10.129.2.70:8443 150m
token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
oc exec ansiblemetrics-controller-manager-<id> -- curl -k -H "Authoriza tion: Bearer USDtoken" 'https://10.129.2.70:8443/metrics' | grep my_counter
HELP my_counter_metric Add 3.14 to the counter TYPE my_counter_metric counter my_counter_metric 2
oc exec ansiblemetrics-controller-manager-<id> -- curl -k -H "Authoriza tion: Bearer USDtoken" 'https://10.129.2.70:8443/metrics' | grep gauge
HELP my_gauge_metric Create my gauge and set it to 2.
oc exec ansiblemetrics-controller-manager-<id> -- curl -k -H "Authoriza tion: Bearer USDtoken" 'https://10.129.2.70:8443/metrics' | grep Observe
HELP my_histogram_metric Observe my histogram HELP my_summary_metric Observe my summary
import ( "github.com/operator-framework/operator-sdk/pkg/leader" ) func main() { err = leader.Become(context.TODO(), "memcached-operator-lock") if err != nil { log.Error(err, "Failed to retry for leader lock") os.Exit(1) } }
import ( "sigs.k8s.io/controller-runtime/pkg/manager" ) func main() { opts := manager.Options{ LeaderElection: true, LeaderElectionID: "memcached-operator-lock" } mgr, err := manager.New(cfg, opts) }
cfg = Config{ log: logf.Log.WithName("prune"), DryRun: false, Clientset: client, LabelSelector: "app=<operator_name>", Resources: []schema.GroupVersionKind{ {Group: "", Version: "", Kind: PodKind}, }, Namespaces: []string{"default"}, Strategy: StrategyConfig{ Mode: MaxCountStrategy, MaxCountSetting: 1, }, PreDeleteHook: myhook, }
err := cfg.Execute(ctx)
packagemanifests/ └── etcd ├── 0.0.1 │ ├── etcdcluster.crd.yaml │ └── etcdoperator.clusterserviceversion.yaml ├── 0.0.2 │ ├── etcdbackup.crd.yaml │ ├── etcdcluster.crd.yaml │ ├── etcdoperator.v0.0.2.clusterserviceversion.yaml │ └── etcdrestore.crd.yaml └── etcd.package.yaml
bundle/ ├── bundle-0.0.1 │ ├── bundle.Dockerfile │ ├── manifests │ │ ├── etcdcluster.crd.yaml │ │ ├── etcdoperator.clusterserviceversion.yaml │ ├── metadata │ │ └── annotations.yaml │ └── tests │ └── scorecard │ └── config.yaml └── bundle-0.0.2 ├── bundle.Dockerfile ├── manifests │ ├── etcdbackup.crd.yaml │ ├── etcdcluster.crd.yaml │ ├── etcdoperator.v0.0.2.clusterserviceversion.yaml │ ├── etcdrestore.crd.yaml ├── metadata │ └── annotations.yaml └── tests └── scorecard └── config.yaml
operator-sdk pkgman-to-bundle <package_manifests_dir> \ 1 [--output-dir <directory>] \ 2 --image-tag-base <image_name_base> 3
operator-sdk run bundle <bundle_image_name>:<tag>
INFO[0025] Successfully created registry pod: quay-io-my-etcd-0-9-4 INFO[0025] Created CatalogSource: etcd-catalog INFO[0026] OperatorGroup "operator-sdk-og" created INFO[0026] Created Subscription: etcdoperator-v0-9-4-sub INFO[0031] Approved InstallPlan install-5t58z for the Subscription: etcdoperator-v0-9-4-sub INFO[0031] Waiting for ClusterServiceVersion "default/etcdoperator.v0.9.4" to reach 'Succeeded' phase INFO[0032] Waiting for ClusterServiceVersion "default/etcdoperator.v0.9.4" to appear INFO[0048] Found ClusterServiceVersion "default/etcdoperator.v0.9.4" phase: Pending INFO[0049] Found ClusterServiceVersion "default/etcdoperator.v0.9.4" phase: Installing INFO[0064] Found ClusterServiceVersion "default/etcdoperator.v0.9.4" phase: Succeeded INFO[0065] OLM has successfully installed "etcdoperator.v0.9.4"
operator-sdk <command> [<subcommand>] [<argument>] [<flags>]
operator-sdk completion bash
bash completion for operator-sdk -*- shell-script -*- ex: ts=4 sw=4 et filetype=sh