Project: Botkube (Observability and Analysis / Observability)
File: 2022-06-15-cfg-syntax-issues.md

Created on 2022-06-15 by Mateusz Szostok () This document describes found issue with the current syntax for Botkube configuration. The communications settings are stored in the `comm_config.yaml` file. All communicator settings are in one YAML. It contains communicators but also "sinks" like Elasticsearch. They are a bit different. The communicators are bidirectional while ES is only able to receive events. ```yaml communications: slack: enabled: false channel: 'SLACK_CHANNEL' token: 'SLACKAPITOKEN' elasticsearch: enabled: false server: 'ELASTICSEARCH_ADDRESS' # e.g https://example.com:9243 username: 'ELASTICSEARCH_USERNAME' # Basic Auth index: name: botkube type: botkube-event shards: 1 replicas: 0 ``` Split into messaging programs and "sinks". The resource settings are stored in the `resource_config.yaml` file. The resource configuration file contains: Notification settings about K8s events Kubectl command executor settings Global settings like cluster name, upgrade notifications settings, etc. Information if recommendations should be enabled One huge YAML that you need to scroll. It holds too much different configuration. You are not able to enable/disable a given recommendation. There is only the `recommendations: true` property. You are not able to manage filters settings. There are no property for that. The `kubectl` executor settings are under `settings`. No option to define multiple notification settings. Currently, you need to deploy Botkube twice if you want to have two different notification strategies. Extract `settings.kubectl` to `executors[].kubectl`. In the future we will add more executors there, e.g. `helm`, `istioctl` etc. Nest `resources` under `notifiers[].kubernetes.resources`. In the future we will add more platforms that will send events, e.g. Sysdig, KubePug, etc. Notifiers <table> <tr> <td> Before </td> <td> After </td> </tr> <tr> <td> ```yaml resources: name: v1/nodes namespaces: include: all events: error recommendations: true ``` </td> <td> ```yaml notifiers: kubernetes: resources: name: v1/nodes namespaces: include: all events: error sysdig: kubePug: recommendations: image: # "Checks and adds recommendation if 'latest' image tag is used for container image." enabled: true pod: # "Checks and adds recommendations if labels are missing in the pod specs." enabled: true ingress: # "Checks if services and tls secrets used in ingress specs are available." enabled: true ``` </td> </tr> </table> Executors <table> <tr> <td> Before </td> <td> After </td> </tr> <tr> <td> ```yaml settings: clusterName: not-configured kubectl: enabled: false commands: verbs: ["api-resources", "...", "auth"] resources: ["deployments", "...", "nodes"] ``` </td> <td> ```yaml executors: kubectl: enabled: false commands: verbs: ["api-resources", "...", "auth"] resources: ["deployments", "...", "nodes"] helm: istioctl: ``` </td> </tr> </table> Botkube settings: Stay as they are right now. ```yaml settings: clusterName: not-configured configWatcher: true upgradeNotifier: true ``` The API is cleaner, but we still need to be able to configure a given "notifier/executor" multiple times. Let's introduce . Notifiers <table> <tr> <td> Before </td> <td> After </td> </tr> <tr> <td> ```yaml notifiers: kubernetes: resources: name: v1/nodes namespaces: include: all events: error sysdig: kubePug: ``` </td> <td> ```yaml notifiers: name: nodes-errors # name used for bindings kubernetes: resources: name: v1/nodes namespaces: include: all events: error sysdig: kubePug: ``` </td> </tr> </table> <details> <summary>Discarded alternative</summary> ```yaml notifiers: kubernetes: name: nodes-errors # name used for bindings resources: name: v1/nodes namespaces: include: all events: error ``` </details> Executors <table> <tr> <td> Before </td> <td> After </td> </tr> <tr> <td> ```yaml executors: kubectl: enabled: false commands: verbs: ["api-resources", "...", "auth"] resources: ["deployments",

"nodes"] helm: istioctl ``` </td> <td> ```yaml executors: name: kubectl-read-only # name used for bindings kubectl: namespaces: include: team-a commands: verbs: ["api-resources", "...", "auth"] resources: ["deployments", "..", "nodes"] name: helm-full-access # name used for bindings helm: namespaces: include: team-a commands: verbs: ["list", "delete", "install"] ``` </td> </tr> </table> Extend each "communication" platform with dedicated bindings: ```yaml communications: # having multiple slacks? or ES? name: tenant-b-workspace slack: token: 'SLACKAPITOKEN' channels: name: "#team-a" bindings: notifiers: "nodes-errors" "depreacted-api" executors: "kubectl-read-only" "helm-full-access" ``` See the proposal. Introduce "profiles/policies/presets" that can gather the given configuration together. ```yaml communications: slack: token: 'SLACKAPITOKEN' notification: type: short policyBinding: channel: "#dev" policies: development channel: "#prod" policies: production admin channel: "#admin" policies: admin ``` See the investigation. You are not able to "disable/enable" them via config. It needs to be done via command. The filter package holds not only functionality to "filter" objects but also to mutate or validate them. For example: | Name | Type | Description | |-|--|| | Object Annotation Checker | mutator | Checks if object has `"botkube.io/channel"` and if yes, change the default channel where notification will be sent. | | IngressValidator | validator | "Checks if services and tls secrets used in ingress specs are available." | | NamespaceChecker | filter | "Checks if event belongs to blocklisted namespaces and filter them." | Separate them into `Filters`, `Mutators`, `Validators` - this can be too complex for now. or rename `Filters` to `Mutators` and leave there only this functionality that mutates state. Rest is mostly about recommendation, so belongs to `notifiers`. However, I don't see any candidate for `Mutators` right now. | Filter Name | Description | Note | |-|--|--| | ImageTagChecker | Checks and adds recommendation if 'latest' image tag is used for container image. | Move as recommendation notificator. | | IngressValidator | Checks if services and tls secrets used in ingress specs are available. | Move as recommendation notificator. | | ObjectAnnotationChecker | Checks if annotations botkube.io/* present in object specs and filters them. | Remove it. | | PodLabelChecker | Checks and adds recommendations if labels are missing in the pod specs. | Move as recommendation notificator. | | NamespaceChecker | Checks if event belongs to blocklisted namespaces and filter them. | Remove it. It will be per resource now. | | NodeEventsChecker | Sends notifications on node level critical events. | Move as K8s events notificator. | As a result, we only need to move all functionality under `notifier` property. Issues that are still not addressed: Showing status of a given extension - if it's up and running. Currently, there is no feedback channel. We can check that only in Botkube logs. Providing metadata information about given extension (icon, display name, docs url etc.). Will be useful for discoverability. Currently, not available. Out-of-the-box validation via Open API schema. Currently, not available. Easy extensibility - add a new executor/notificator. Currently, via built-in filters. Those issues can be address with dedicated Botkube configuration CRDs. See investigation. Even though the option to seems to be more flexible, I think that it's too big to be implemented at the current stage of the Botkube. I propose to solve the syntax issues and multichannel feature as described in the proposal.