Project: Botkube (Observability and Analysis / Observability)
File: 2022-06-20-cfg-via-crds.md

Created on 2022-06-20 by Mateusz Szostok () Note It was replaced by the proposal. See the section to learn more. <!-- toc --> * + + * + * * + <!-- tocstop --> This document is based on the outcome from document. It describes the possible solution to configure Botkube via dedicated CustomResources (CR) instead of configuration files. It also addresses the multichannel support. Make Botkube, Kube native. Defining configuration via CRD allows: Easy extensibility - new executor/notificator can be dynamically created, and it will be automatically picked by controller. Currently, two big YAML file, requires Botkube restart, extensions need to be built-in. Showing status of a given extension - if it's up and running. ```yaml status: phase: Initializing/Connected/Failed message: "connection failed: 401 Unautorized" ``` Currently, no feedback channel - only container logs. Providing metadata information about given extension (icon, display name, docs url etc.). Will be useful for discoverability. Currently, not available. Out-of-the-box validation via Open API schema. Currently, not available. Configuration that will be cluster-wide and namespace-scoped. Currently, it's possible but not in a native K8s way. However, switching from config files to CRDs also adds some limitations: 1MB size limit per configuration definition. Configuration is purly K8s based, so it means no option to run on bare-metal/docker/etc. Currently, we have two config YAMLs that can be configured. Namespace-scoped CR will make it harder. For example, you need to ensure that Secret with a communicator token is in all Namespaces where communication CR was created. It has pros and cons. A pros are definitely a better security model and fine-grained approach. > REMEDIATION: To simplify transition, we can start with the cluster-wide CR, that will define globally settings - same as current YAML files. Later we can introduce namespaced version to allow fine-grained configuration. Domains: Communicators (Cluster)CommunicatorTemplate (Cluster)Slack/(Cluster)Discord/(Cluster)MSTeams/etc. Executors (Cluster)ExecutorTemplate (Cluster)Executor Notifiers (including validators) (Cluster)NotifierTemplate (Cluster)Notifier Mutators (filters) (Cluster)MutatorTemplate (Cluster)Mutator Initially, all executors and notifiers can be marked as built-in. The `spec.plugin.built-in: {name}` marks that a given functionality is built-in. We can later extract it into separate plugin (probably Docker image). example on how it can be consumed on Dashboard ```yaml apiVersion: "core.botkube.io/v1" kind: ClusterExecutorTemplate metadata: name: Kubectl spec: validation: openAPIV3Schema: type: object title: "Parameters for kubectl executor" required: [ "enabled", "commands" ] properties: enabled: title: "Set true to enable kubectl commands execution" default: true type: boolean commands: type: object required: [ "verbs", "resources" ] properties: verbs: title: "Methods which are allowed" default: [ "api-resources", "api-versions" ] type: array items: type: string enum: [ "api-resources", "api-versions", "cluster-info", "describe", "diff", "explain", "get", "logs", "top", "auth" ] resources: title: "Resource configuration which is allowed" type: array default: [ "deployments","pods" ] items: type: string enum: [ "deployments", "pods" , "namespaces", "daemonsets", "statefulsets", "storageclasses", "nodes" ] plugin: built-in: kubectl # built-in name, so we know which one to pick. we can also use 'metadata.name'. metadata: displayName: Execute Kubectl CLI license: name: "MIT" documentationURL: https://examples.com/docs supportURL: https://example.com/online-support iconURL: https://examples.com/favicon.ico maintainers: email: foo@example.com name: Foo Bar url: https://examples.com/foo/bar status: phase: Registered/Failed ``` example on how it can be consumed on Dashboard ```yaml apiVersion: "core.botkube.io/v1" kind: ClusterExecutor metadata: name: kubectl-readonly spec: template: Kubectl parameters: namespaces: include: ".*" commands: verbs: ["get", "logs"] resources: ["Deployments", "Pods", "Services"] apiVersion:

kind: Executor metadata: name: kubectl-readonly namespace: team-a spec: template: Kubectl parameters: commands: verbs: ["get", "logs"] resources: ["Deployments", "Pods", "Services"] ``` ```yaml apiVersion: "core.botkube.io/v1" kind: (Cluster)NotifierTemplate metadata: name: Kubernetes spec: validation: openAPIV3Schema: properties: type: object plugin: built-in: k8s # needed for the first phase, when we don't want to extract all logic into separate plugins/Docker images. metadata: displayName: Notify about Kuberentes events. license: name: "MIT" documentationURL: https://examples.com/docs supportURL: https://example.com/online-support iconURL: https://examples.com/favicon.ico maintainers: email: foo@example.com name: Foo Bar url: https://examples.com/foo/bar status: phase: Registered/Failed ``` ```yaml apiVersion: "core.botkube.io/v1" kind: ClusterNotifier metadata: name: k8s-network-errors spec: template: Kubernetes # name of ClusterNotifierTemplate parameters: # Core webhook for ClusterNotifier, validates it against 'ClusterNotifierTemplate[Kubernetes].validation.openAPIV3Schema' namespaces: # global, can be OVERRIDDEN per resource include: ".*" resources: name: v1/pods namespaces: # it overrides the top one! include: istio-system events: error name: v1/services events: error name: networking.istio.io/v1alpha3/DestinationRules events: error name: networking.istio.io/v1alpha3/VirtualServices events: error status: phase: Initializing/Serving/Failed apiVersion: "core.botkube.io/v1" kind: Notifier metadata: name: k8s-network-errors namespace: team-a # only here Namespace is allowed. All events are scoped to this one. spec: template: Kubernetes # refers Notifier not ClusterNotifier parameters: resources: name: v1/pods events: error name: v1/services events: error name: networking.istio.io/v1alpha3/DestinationRules events: error name: networking.istio.io/v1alpha3/VirtualServices events: error ``` Communicators integration in Botkube is quite narrow in comparison to executors or notifiers, and it's not the main extension part. We can even decide to represent is as fixed set of CRDs, see . In the first implementation also the Namespace-scoped CRD doesn't make sens. ```yaml apiVersion: "core.botkube.io/v1" kind: ClusterCommunicatorTemplate metadata: name: Slack spec: validation: openAPIV3Schema: properties: type: object plugin: built-in: slack # needed for the first phase, when we don't want to extract all logic into separate plugins/Docker images. metadata: displayName: Slack communicator description: Connector for Slack communicator that helps to monitor your Kubernetes cluster, debug deployments and run custom checks on resource specs. license: name: "MIT" documentationURL: https://examples.com/docs supportURL: https://example.com/online-support iconURL: https://examples.com/favicon.ico maintainers: email: foo@example.com name: Foo Bar url: https://examples.com/foo/bar status: phase: Registered/Failed message: "CRD 'clusterslack.communicators.core.botkube.io/v1' not registered in cluster." ``` ```yaml apiVersion: "core.botkube.io/v1" kind: ClusterCommunicator metadata: name: slack-instance spec: template: Slack parameters: notification: type: short token: valueFrom: secretKeyRef: name: communication-slack namespace: botkube-system key: token channels: name: nodes # it's better with string as YAML doesn't support #, or @ chars bindings: notifier: name: nodes-errors executors: name: nodes-readonly status: phase: Initializing/Connected/Failed message: "connection failed: 401 Unautorized" ``` Currently, I don't see any sensible candidates for this kind. The ObjectAnnotationChecker and NamespaceChecker are redundant if new syntax will be implemented. In my opinion, it's better to remove them to simplify the configuration settings. However, to do it right, first we should add a new metric to check how many users are using it. | Filter Name | Description | Note | |-|--|--| | ImageTagChecker | Checks and adds recommendation if 'latest' image tag is used for container image. | Move as recommendation notificator. | | IngressValidator | Checks if services and tls secrets used in ingress specs are available. | Move as recommendation notificator. | | ObjectAnnotationChecker | Checks if annotations botkube.io/* present in object specs and filters them. | Remove it. | | PodLabelChecker | Checks and adds recommendations if labels are missing in the pod specs. | Move as recommendation notificator. | | NamespaceChecker | Checks if event belongs to blocklisted namespaces and filter them. | Remove it. It will be per resource now. | | NodeEventsChecker | Sends notifications on node level critical events. | Move as K8s events

| This section described necessary changes if the implementation idea will be accepted and planned in the roadmap. Add new CRDs with a new configuration syntax. Create cluster-wide controllers for: Communicators - built-in ClusterCommunicatorTemplate ClusterCommunicator Executors - built-in ClusterExecutorTemplate ClusterExecutor Notifiers - built-in ClusterNotifierTemplate ClusterNotifier All communicators, executors, notifiers are still built-in. Update documentation about configuration. Deprecate usage of the `resourceconfig.yaml` and `commconfig.yaml` config files. Create Namespace-scoped controllers for: Communicators CommunicatorTemplate Communicator Executors ExecutorTemplate Executor Notifiers NotifierTemplate Notifier Recommendations are merged under notifiers. Filters are removed and existing one are moved under notifiers. Update `@Botkube` commands to reflect new configuration. Add option to manage all settings via communicator. GraphQL gateway service that will expose CRs. Used by Dashboard. Optional: Add CLI to simplify creating/updating configuration. The CRD seems to be the most flexible and Kubernetes native approach for handling Botkube configuration. It enables a lot of new features. However, we still don't know the future of the Botkube and how it will be used. Because it is a huge change and the implementation phase will be time-consuming, I propose to postpone it until we will know answers for questions like: what is the desired way of extending Botkube? what is the main extensions part? Notifiers, Communicators, Executors or something else? what are the requirements regarding Botkube dashboard? what features we want to support? assign predefined action buttons? customizable message format? interactive recommendations? Other approaches that I consider with explanation why I ruled them out. <details> <summary>Discarded alternative</summary> Each communicator, executor, and notificator is represented by own CRD. Communicators integration in Botkube is quite narrow in comparison to executors or notifiers, and it's not the main extension part. To simplify the Botkube implementation we can still have them as built-in. To still be K8s native, we can define them as CRDs. For example: `Slack/ClusterSlack.communicators.core.botkube.io/v1` `Discord/ClusterDiscord.communicators.core.botkube.io/v1` `Mattermost/ClusterMattermost.communicators.core.botkube.io/v1` `MSTeams/ClusterMSTeams.communicators.core.botkube.io/v1` `Elasticsearch/ClusterElasticsearch.communicators.core.botkube.io/v1` ```yaml apiVersion: "core.botkube.io/v1" kind: (Cluster)CommunicatorTemplate metadata: name: slack-instance spec: crd: name: ClusterSlack.communicators.core.botkube.io/v1 metadata: name: Slack displayName: Slack communicator description: Connector for Slack communicator that helps to monitor your Kubernetes cluster, debug deployments and run custom checks on resource specs. license: name: "MIT" documentationURL: https://examples.com/docs supportURL: https://example.com/online-support iconURL: https://examples.com/favicon.ico maintainers: email: foo@example.com name: Foo Bar url: https://examples.com/foo/bar status: phase: Registered/Failed message: "CRD 'clusterslack.communicators.core.botkube.io/v1' not registered in cluster." apiVersion: "communicators.core.botkube.io/v1" kind: ClusterSlack metadata: name: slack-instance spec: notification: type: short token: valueFrom: secretKeyRef: name: communication-slack namespace: botkube-system key: token channels: name: nodes # it's better with string as YAML doesn't support #, or @ chars bindings: notifier: kind: ClusterKubernetes name: nodes-errors executors: kind: ClusterKubectl name: nodes-readonly status: phase: Initializing/Connected/Failed message: "connection failed: 401 Unautorized" ``` ```yaml apiVersion: "executors.core.botkube.io/v1" kind: ClusterKubectl metadata: name: kubectl-readonly spec: namespaces: include: ".*" commands: verbs: ["get", "logs"] resources: ["Deployments", "Pods", "Services"] apiVersion: "executors.core.botkube.io/v1" kind: Kubectl metadata: name: kubectl-readonly namespace: team-a spec: commands: verbs: ["get", "logs"] resources: ["Deployments", "Pods", "Services"] ``` ```yaml apiVersion: "notifier.core.botkube.io/v1" kind: Kubernetes metadata: name: network-errors spec: namespaces: # global, can be OVERRIDDEN per resource include: ".*" resources: name: v1/pods namespaces: # it overrides the top one! include: istio-system events: error name: v1/services events: error name: networking.istio.io/v1alpha3/DestinationRules events: error name: networking.istio.io/v1alpha3/VirtualServices events: error apiVersion: "notifier.core.botkube.io/v1" kind: Kubernetes metadata: name: network-errors namespace: team-a # only here Namespace is allowed. All events are scoped to this one. spec: resources: name: v1/pods events: error name: v1/services events: error name: networking.istio.io/v1alpha3/DestinationRules events: error name: networking.istio.io/v1alpha3/VirtualServices events: error ``` </details>