Project: Botkube (Observability and Analysis / Observability)
File: 2022-12-23-rbac.md

Created on 2022-12-23 by Pawe Kosiec () | Status | | - | | `PROPOSED` | Botkube exposes an ability to run arbitrary commands, especially those related to Kubernetes clusters. Currently, to execute any `kubectl` command, Botkube uses ClusterRole which comes with an installation. This ClusterRole is bound to a ServiceAccount used by a Botkube Pod. This raises security concerns. Also, while Botkube supports per-channel kubectl execution configuration, users should be allowed to set per-user or group mapping for more complex scenarios. While initially we could stick to channel-based grouping, it would be great to have an extensible mechanism to cover future use cases. <!-- toc --> - - - - - - <!-- tocstop --> Stop using ClusterRole assigned to the Botkube pod for executing commands. Implement ability to configure command execution permissions for different user groups. Add ability to map users from the communication platforms to Kubernetes permissions. Ensure that the proposed approach can be used with external identity providers. Support plugins. RBAC support is plugin-specific. Not all executors are based on Kubernetes API. This section describes how RBAC will be implemented for executors. To identify users and groups, we have multiple options, which varies between communication platforms. Summarizing the options, the proposal suggests the following approaches: User identification: User email Static value for a given configuration Group identification: Channel name User groups (Slack, Mattermost), Roles (Discord). Not supported by MS Teams. Static value for a given configuration We can support just a few options initially, and later add more based on user feedback. See the paragraph for more details. To get user email, channel name or user group name, we would need to do additional API calls in bot logic. However, we can cache the fetched values. Slack: We need to add two new scopes to the Slack app: `users:read` and `users:read.email` We need to do additional API calls to get email (user info for a given ID) and channel name (conversation info for a given ID) Unfortunately Slack user groups are in paid plan, so I couldn't test them. Probably we will need one additional call to get user group details to get its name. Mattermost: We need to do additional API call to get user email based on its ID . They are not included in the message data. Discord: We need to query user data as they are missing from the message object. Quoting the docs: "The field user won't be included in the member object attached to MESSAGECREATE and MESSAGEUPDATE gateway events." . ```go &discordgo.MessageCreate{ Message: &discordgo.Message{ // ... Member: &discordgo.Member{ // ... Roles: []string{ "976789858670497805", // <-- this is what we can use }, }, }, } ``` Based on that we can query for a full role object to get its name. ```go b.api.AddHandler(func(s discordgo.Session, m discordgo.MessageCreate) { msg := discordMessage{ Event: m, } roles, _ := b.api.GuildRoles(m.GuildID) // manually filter role by ID - unfortunately there's just list capability - couldn't find get // we can cache the roles as they are server-wide (or guild-wide, depends what term is used) // get role.Name - more: https://discord.com/developers/docs/topics/permissions#role-object ``` MS Teams: We need to do additional API calls to get the user name email. Firstly, we need to , then . There are no user groups or roles in MS Teams we can use. Cluster Admin creates Roles and ClusterRoles, as well as the RoleBindings and ClusterRoleBindings. For example: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: read-only subjects: kind: Group # group binding name: developers apiGroup: rbac.authorization.k8s.io kind: User # user binding name:

apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: read-only apiGroup: rbac.authorization.k8s.io ``` Next, Botkube needs to be aware of such configuration. In order to achieve this, we'll extend our execution configuration. We would support multiple types of mapping: user-email based (allows per-user access rights configuration) channel-based (allows per-channel access rights configuration) user-group based (for supported platforms we'll use user groups or roles; for MS Teams, we'll fallback to channel-based mapping) static (allows to impersonate a given user or group every time a given executor is executed regardless the context) NOTE: Username and groups prefix could be useful in the clusters where OIDC is configured. For example, see blog post. ```yaml executors: 'kubectl-read-only': botkube/kubectl: enabled: true context: rbac: &rbacCtx user: type: Email # user email will be used as identifying subject without group prefix: "" # prefix added to the user email fetched from comm platform group: type: "" # equal to `type: Disabled` - no group mapping prefix: "" user: type: Email # user email will be used as user name prefix: "" # prefix added to the user email fetched from comm platform group: type: ChannelName # channel name will be used as identifying subject prefix: "" # added to the channel name user: type: Email # user email will be used as user name prefix: "" # prefix added to the user email fetched from comm platform group: type: UserGroupName # channel name will be used as identifying subject prefix: "" # added to the user group name user: type: Static # impersonate as a given user every time a given executor is executed static: value: "default" group: # Optional, if a static group impersonation should be used type: Static # impersonate as a given group every time a given executor is executed static: value: [ "developers" ] # groups to impersonate config: defaultNamespace: default restrictAccess: false 'kubectl-pods-rw': botkube/kubectl: enabled: true context: rbac: &rbacCtx # no need to specify it again if it's the same as above 'kubectl-deploy-rw': botkube/kubectl@v1.0.0: enabled: true context: rbac: &rbacCtx # no need to specify it again if it's the same as above ``` A few remarks: Merging different mapping configuration won't besupported and will result in an error. The configuration will be validated during Botkube startup. As Kubernetes impersonation requires an username when using group impersonation, the default user name for mapping will be set as a static `default` value, as shown in Option 4 above. If the `rbac` context is not provided (as a result, `rbac.user|group.type` is empty), Botkube won't create a temporary Kubeconfig for a given plugin. By default, the static mapping will be used, to keep the current Botkube behavior. During Botkube installation, we will create ClusterRole and ClusterRoleBinding resources for it. If users would like to use channel-based access rights, they will need to create Kubernetes resources manually and configure mappings in Botkube configuration. At later point we can think of facilitating access rights configuration by managing Roles, ClusterRoles, ClusterRoleBindings and RoleBindings directly from Botkube. To ensure compatibility with our plugin system, we'll use to create a special short-living Kubeconfig to use for a given execution. This will be handled globally by Botkube for all plugins. We'll extend our gPRC API for executors in the following way: ```proto message ExecuteRequest { // Commands represents the exact command that was specified by the user. string command = 1; // Configs is a list of Executor configurations specified by

repeated Config configs = 2; // New field ExecuteContext context = 3; } message ExecuteContext { string kubeconfig_path = 1; } ``` This path may be used by the plugin to access Kubernetes cluster in a restricted way. Later we can introduce more fields to the context, such as user details extracted for a given command execution. To ensure no plugin has access to the sensitive configuration data, like Slack or Discord tokens, we will modify the plugin manager to run command with in an isolated environment in the following way. For each executor plugin, We will use chroot to limit the access to the filesystem. A given plugin will have an isolated directory with all dependencies (e.g. `helm` or `kubectl` binary). For each command execution Botkube plugin manager will: create a temporary subdirectory inside the isolated directory, create a temporary Kubeconfig file in the subdirectory, pass the Kubeconfig path via gRPC API, delete the temporary subdirectory after command execution. This is possible as we have full control over how the command is run: ```go for key, path := range bins { // ... //nolint:gosec // warns us about 'Subprocess launching with variable', but we are the one that created that variable. cmd := exec.Command(path) cmd.SysProcAttr = &syscall.SysProcAttr{ Chroot: tmpExecutionDir, // change root directory Credential: &syscall.Credential{ // change user / group - if we considered to have also another user/group for plugins Uid: ..., Gid: ..., }, } cli := plugin.NewClient(&plugin.ClientConfig{ Cmd: cmd, // ... }) // ... } ``` For such temporary Kubeconfig we still use the same token as the Botkube installation. That means, user needs to trust each plugin before execution, as it can behave maliciously, and e.g. impersonate a `cluster-admin` with a modified Kubeconfig. Fortunately, Cluster Admin can prevent this, by defining tighter Botkube ClusterRole: ```yaml apiGroups: [""] resources: ["groups"] verbs: ["impersonate"] resourceNames: ["developers", "testers"] # group names apiGroups: [""] resources: ["users"] verbs: ["impersonate"] resourceNames: ["dev@example.com"] # user names ``` Still, a harmful plugin can use any of the group names / users defined in the ClusterRole (if it somehow guesses them) and we cannot prevent this, if we pass the token with ability to impersonate. For source plugins, we will use the same approach as for executors. NOTE: Before you read this paragraph, get familiar with section. Cluster Admin creates Roles, ClusterRoles, RoleBindings and ClusterRoleBindings. Then, to configure the mapping, Cluster Admin uses the same syntax as for Executor plugins: ```yaml sources: 'plugin-based': botkube/cm-watcher: enabled: true context: rbac: user: type: Static # impersonate as a given user when starting source plugin static: value: "default" group: type: ChannelName # channel name will be used as identifying subject prefix: "" # added to the channel name user: type: Static # impersonate as a given user when starting source plugin static: value: "default" group: type: Static # impersonate as a given group when starting source plugin static: value: [ "developers" ] # groups to impersonate ``` Initially, we will support only static mapping, with the channel-based group mapping as a nice-to-have feature. Same as for Executors, we will use Kubernetes user impersonation to create Kubeconfig for each source plugin startup. This will be handled globally by Botkube for all plugins. We'll extend our gPRC API for sources in the following way: ```proto message StreamRequest { // Configs is a list of Source configurations specified by users. repeated Config configs = 1; // New field SourceContext context = 3; } message SourceContext { string kubeconfig_path = 1; } ``` This path may be used by the plugin to access Kubernetes cluster in a restricted way. The source plugin isolation is the same as for

We will run the subprocess in an isolated environment, with a Kubeconfig file per each configuration. See the to understand what code changes are needed to implement the production solution. This section covers all alternative solutions that were considered during the design process. Initial mapping config was proposed in the following way: ```yaml executors: 'kubectl-read-only': botkube/kubectl: enabled: true context: kubeconfig: &kubeconfigCtx mapping: type: UserEmail # user email will be used as identifying subject without group usernamePrefix: "" # prefix added to the user email fetched from comm platform groupsPrefix: "" # not used mapping: type: ChannelName # channel name will be used as identifying subject usernamePrefix: "" # prefix added to the user email fetched from comm platform groupsPrefix: "" # added to the channel name mapping: type: UserGroupName # user group name will be used as identifying subject usernamePrefix: "" # prefix added to the user email fetched from comm platform groupsPrefix: "" # added to the user group name mapping: type: Static # impersonate as a given group or user every time a given executor is executed username: "any" # username to impersonate groups: [ "developers" ] # groups to impersonate config: defaultNamespace: default restrictAccess: false 'kubectl-pods-rw': botkube/kubectl: enabled: true context: kubeconfig: &kubeconfigCtx # no need to specify it again if it's the same as above 'kubectl-deploy-rw': botkube/kubectl@v1.0.0: enabled: true context: kubeconfig: &kubeconfigCtx # no need to specify it again if it's the same as above ``` However, allowing mapping user and group separately gives us more flexibility. For example, we will be able to implement username mapping for users (for those who don't want to share email addresses with Botkube), and still getting group name from the channel name. After identifying the user and/or group, instead of introspection, we could run executor as a short-living separate Pod with a dedicated ServiceAccount and (Cluster)RoleBinding. This way we could use Kubernetes RBAC to restrict access to the cluster, with the isolation provided by Kubernetes. However, this is not possible as our plugin system is not compatible: While the plugin system is over RPC, it is currently only designed to work over a local reliable network. Plugins over a real network are not supported and will lead to unexpected behavior. Source: https://github.com/hashicorp/go-plugin Once the proposal is accepted, the following changes will be made: Update configuration: Introduce `executorscontext` and `sourcecontext` fields. Initially support for executors: static mapping for both users and groups, channel-based group mapping. Create a follow-up task to support additional mappings and monitor the community demand for it: email-based and username-based user mapping, user-group-based group mapping. For sources, initially support static user and group mapping. Nice to have: chabnel-based group mapping if the implementation won't be hard. Validate executor and source bindings configuration and return error if kubeconfig RBAC mapping is different for the same bound plugins. Update defaults to use static mapping. Enable creating optional ClusterRole for all users to keep previous behavior. Allow restricting access to specific groups and/or users for impersonation in Helm chart. Handle Kubeconfig passing for executors: Generate and pass Kubeconfig as a part of Context for each execution of a Kubernetes-related executor. Modify plugin manager to run plugins in a restricted environment. Refactor `kubectl` executor: Remove property and our custom policy engine Ensure the `kubectl auth can-i` command can be executed from a communication platform. This can be done as a part of issue. Refactor Kubernetes source: Ensure we use proper Kubeconfig passed via gRPC API. This can be done as a part of issue. Update documentation with the latest RBAC changes: Add required scopes to the communication apps (e.g. Slack) to identify