Project: Teleport (Provisioning / Security & Compliance)
File: 0043-kubeaccess-multiparty.md

authors: Joel Wejdenstl (jwejdenstal@goteleport.com) state: implemented Implement the ability to define RBAC policies with filters similar to those of that can be used to require others to be present in a session for it to be usable. Heavily regulated and security-critical industries require that one or more participants with a certain role are present in SSH and Kubernetes sessions and viewing it live to guarantee that the operator does not perform any mistakes or acts of malice. Such participants need to have the power to terminate a session immediately should anything go wrong. To suit everyone this will need a more detailed configuration model based on rules that can be used to define observers, their powers, and when and in what capacity they are required. This RFD repeatedly refers to some specific nouns of which explanations can be found below: Participant: Any user in the session regardless of mode, including the session initiator. Initiator: The user that started the session. This user is in peer mode. Mode: One of the modes below. Any given participant must be in one mode only. Observer: A participant who can only view the session. Moderator: A participant who can view and terminate the session at any point. Peer: A participant with the ability to view and interact with the session. SSH sessions via TSH currently have support for sessions with multiple users at once. This concept is to be extended to Kubernetes Access which will allow us to build additional features on top. Multiparty sessions shall be implemented by modifying the k8s request proxy forwarder in the `kubernetes_service`. This approach was chosen as it is a hub that sessions pass through which makes it optimal for multiplexing. An approach using multiplexing in the `proxy_service` layer was considered but was ultimately determined to be more complicated since proxies don't handle the final session traffic hop when using Kubernetes Access. It will be implemented by adding a multiplexing layer inside the forwarder that similar to the current session recording functionality, but instead this multiplexes output to the session initiator and all participants but only streams back input from participants with the `peer` mode. A core feature we need to support is the required participants. This will allow cluster administrators to configure policies that require certain Teleport users of a certain role to be actively monitoring the session. This feature is useful in security-critical environments where multiple people need to witness every action in the event of severe error or malice and can halt any erroneous or malicious action. By default, a `tsh kube exec` and `tsh ssh` request will go through as usual if no policies are defined. If a policy like the one above is defined the session will be put in a pending state until the required viewers have joined. Sessions can have 3 possible states: `PENDING`\ When a session is in a `PENDING` state, the connection to the pod from the proxy has not yet started and all users are shown a default message informing them that the session is pending. `RUNNING`\ A `RUNNING` session behaves like a normal multiparty session. `stdout`, `stdin` and `stdout` are mapped as usual and the pod can be interacted with. `TERMINATED`\ A session becomes `TERMINATED` once the shell spawned inside the pod quits or is forcefully terminated by one of the session participants. All sessions begin in the `PENDING` state and can change states based on the following transitions: When the requirements for present viewers laid out in the role policy are fulfilled, the session transitions to a `RUNNING`

This involves initiating the connection to the pod and setting up the shell. Finally, all clients are multiplexed onto the correct streams as described previously. Only the session initiator can make input, observers are not connected to the input stream and may only view stdout/stderr and terminate the session. When the shell process created on the pod is terminated, the session transitions to a `TERMINATED` state, and all clients are disconnected as per standard `kubectl` behavior. Session observers that are present may at any point decide to forcefully terminate the session. This will instantly disconnect input and output streams to prevent further communication. Once this is done the Kubernetes proxy will send a termination request to the pod session to request it be stopped. If an observer disconnects from the session in a way that causes the policy for required observers to suddenly not be fulfilled, the session will transition back to a `PENDING` state. In this state, input and output streams are disconnected, preventing any further action. Here, the connection is frozen for a configurable amount of time as a sort of grace period. After a grace period has elapsed in a session in a session that previously was in a `RUNNING` state, the session is automatically terminated. This can be canceled by having the required observers join back in which transitions the session back to `RUNNING`. Any participant of the session can terminate the session in the `PENDING` state. This will simply mark the session as terminated and disconnect the participants as no connection to the pod exists at this time. The initial implementation of multiparty sessions for SSH Kubernetes access will only be supported via CLI access for implementation simplicity. Terminating the `tsh kube exec` or `tsh ssh` process that started the session terminates the session. Terminating a participant `tsh` process disconnects the observer from the session and applies relevant state transitions if any. Terminating the session from a moderator `tsh` instance can be done with the T hotkey. Session creation can happen with the existing flow using `kubectl exec` but the wrapper command `tsh kube exec --invite=bob@example.com,eve@foo.net --reason="Need to fix this pod" -- database_pod -- /bin/bash`. This subcommand allows you to invite one or more accounts that will receive a notification saying they are invited. An arbitrary string may also be provided as a reason for the session invite, it could for example be used to say what the purpose of the session is. Kubernetes itself has no concept of multiparty sessions. This means that we cannot easily use its built-in facilities for support session joining. To make this process easier for the user. I propose extending the current `tsh join` command to also work for Kubernetes access in the form of `tsh kube join <session-id>`. This attaches to an ongoing session and displays stdout/stderr. If the standard `persessionmfa` option is enabled for a role then MFA tap input via Yubikey or other is required for the participant to be considered active. This requirement is on an interval of 1 minute. When there are 15 seconds left, an alert is printed to the console. ``` Teleport > Please tap your MFA key within 15 seconds. ``` If the tap is made after the alert, the following message is shown: ``` Teleport > MFA tap received. ``` For TTY-enabled SSH and Kubernetes sessions that require additional participants regardless of mode as per a require policy, Teleport will now inject broadcast messages into the session to notify participants about the state of the session. No messages will be shown in sessions that do not require additional

This is needed so that participants are aware of what is currently happening in the session for security and usability reasons. Each broadcast message is prefixed with `Teleport > ` to indicate that this message is injected by Teleport and not from the originating shell. There are 10 kinds of broadcast messages: `Creating session with uuid <example-uuid>...`: Sent on session creation. `User <user> joined the session.`: Sent when a user joins the session. `User <user> left the session.`: Sent when a user leaves the session. `Connecting to $HOSTNAME over $PROTOCOL`: Sent when the session is launched and thus transferred to a normal shell. `Session closed.`: Sent when the session is closed due to the termination of the shell. `Session terminated by a moderator`: Sent when moderator forcefully terminates a session. `Session paused, waiting for additional participants...`: Sent when a session is paused due to lack of required participants. `Session resumed.`: Sent when a session has the required participants and is resumed. `Please tap your MFA key within 15 seconds.` and `MFA tap received.` are messages shown to moderators in sessions that are MFA-presence enabled. `Session terminated: participant requirements not met`: Sent when a session is terminated due to lack of required participants and when the pause mode is not enabled. The optional flag `--participant-req` may be passed `tsh kube exec` or `tsh ssh` when the session is started to provide a verbose output of the participant requirements. By default, we avoid this because the output is large and may be overwhelming but the option is provided for users who wish to see exactly what the requirements for starting a session are. In this case, the `Session paused, waiting for additional participants...` message will be replaced by the message below. ``` Teleport > Session paused, waiting for additional participants: role-1: one-of: 2x `contains(user.roles, "auditor")` 1x `contains(user.roles, "admin")` role-2: one-of: 1x `contains(user.roles, "cs-overwatch")` ``` This example illustrates how a group of 3 users of which Alice is the initiator and Eve and Ben are two observers start a multiparty session. Below is a series of events that happen that include what each user sees and actions taken. Alice initiates an interactive session to a pod: `tsh kube exec -st --invite=ben@foo.net,eve@foo.net,alice@foo.net redis-bastion /bin/bash` Alice sees: ``` Teleport > Creating session with uuid <example-uuid>... Teleport > User Alice joined the session. Teleport > This session requires additional participants to start... ``` Eve joins the session with `tsh kube join <example-uuid>` and sees: ``` Please tap MFA key to continue... ``` Eve taps MFA Alice and Eve sees : ``` Teleport > Creating session with uuid <example-uuid>... Teleport > User Alice joined the session. Teleport > This session requires additional participants to start... Teleport > User Eve joined the session. ``` Ben joins the session with `tsh kube join <example-uuid>` and sees: ``` Please tap MFA key to continue... ``` Ben taps MFA Alice, Eve, and Ben sees ``` Teleport > Creating session with uuid <example-uuid>... Teleport > User Alice joined the session. Teleport > This session requires additional participants to start... Teleport > User Eve joined the session. Teleport > User Ben joined the session. Teleport > Connecting to redis-bastion over Kubernetes... redis-bastion@localhost $ ``` The connection to the pod is made and each session turns into a normal shell. Shared sessions for Kubernetes access will have support for participant invites and notifications. Ongoing sessions are tracked and can be listed to make it easier to find and join them. Ongoing sessions you have access to view can be listed with `tsh kube

This easily allows eligible participants to find and join a session waiting for participants easily When the `--invite` flag is used with `tsh kube exec`, the invitees are tracked and included in the session resource which allows Teleport clients and plugins to detect notify them. There currently isn't a general-purpose session resource in Teleport that's suitable. therefore I suggest that this shall be added. This will be initially used for tracking Kubernetes sessions but is compatible with all current and future session types. This resource is stored centrally in the backend and is used for storing and tracking metadata of active sessions. Detailed runtime information needed to join such as the TTY size is stored in memory on the multiplexing node. This effectively replaces the resource defined . ```protobuf // SessionSpecV3 is the specification for a live session. message SessionSpecV3 { // SessionID is unique identifier of this session. string SessionID = 1 [ (gogoproto.jsontag) = "session_id,omitempty" ]; // Namespace is a session namespace, separating sessions from each other. string Namespace = 2 [ (gogoproto.jsontag) = "namespace,omitempty" ]; // Type describes what type of session this is. SessionType Type = 3 [ (gogoproto.jsontag) = "type,omitempty" ]; // State is the current state of this session. SessionState State = 4 [ (gogoproto.jsontag) = "state,omitempty" ]; // Created encodes the time at which the session was registered with the auth // server. google.protobuf.Timestamp Created = 5 [ (gogoproto.stdtime) = true, (gogoproto.nullable) = false, (gogoproto.jsontag) = "created,omitempty" ]; // Expires encodes the time at which this session expires and becomes invalid. google.protobuf.Timestamp Expires = 6 [ (gogoproto.stdtime) = true, (gogoproto.nullable) = false, (gogoproto.jsontag) = "expires,omitempty" ]; // AttachedData is arbitrary attached JSON serialized metadata. string AttachedData = 7 [ (gogoproto.jsontag) = "attached,omitempty" ]; // Reason is an arbitrary string that may be used to describe the session and/or it's // purpose. string Reason = 8 [ (gogoproto.jsontag) = "reason,omitempty" ]; // Invited is a list of invited users, this field is interpreted by different // clients on a best-effort basis and used for delivering notifications to invited users. repeated string Invited = 9 [ (gogoproto.jsontag) = "invited,omitempty" ]; // LastActive holds the information about when the session // was last active google.protobuf.Timestamp LastActive = 10 [ (gogoproto.stdtime) = true, (gogoproto.nullable) = false, (gogoproto.jsontag) = "last_active,omitempty" ]; // Hostname is the address of the target this session is connected to. string Hostname = 12 [ (gogoproto.jsontag) = "target_hostname,omitempty" ]; // Address is the address of the target this session is connected to. string Address = 13 [ (gogoproto.jsontag) = "target_address,omitempty" ]; // ClusterName is the name of cluster that this session belongs to. string ClusterName = 14 [ (gogoproto.jsontag) = "cluster_name,omitempty" ]; // Login is the local login/user on the target used by the session. string Login = 15 [ (gogoproto.jsontag) = "login,omitempty" ]; // Participants is a list of session participants. repeated Participant Participants = 16 [ (gogoproto.jsontag) = "participants,omitempty" ]; } // Participant stores information about a participant in the session. message Participant { // ID is a unique UUID of this participant for a given session. string ID = 1 [ (gogoproto.jsontag) = "id,omitempty" ]; // User is the canonical name of the Teleport user controlling this participant. string User = 2 [ (gogoproto.jsontag) = "user,omitempty" ]; // LastActive is the last time this party was active in the session. string LastActive = 3 [ (gogoproto.jsontag) = "id,omitempty" ]; } // SessionType encodes different types of sessions. enum SessionType { // SessionTypeNone is a placeholder variant and isn't valid. SessionTypeNone = 0; // SessionTypeKubernetes means a session initiated via Kubernetes

SessionTypeKubernetes = 1; // SessionTypeSSH means a standard SSH session initiated via `tsh` or web. SessionTypeSSH = 2; } // SessionState represents the state of a session. enum SessionState { // Pending variant represents a session that is waiting on participants to fulfill the criteria // to start the session. SessionStatePending = 0; // Running variant represents a session that has had its criteria for starting // fulfilled at least once and has transitioned to a RUNNING state. SessionStateRunning = 1; // Terminated variant represents a session that is no longer running and due for removal. SessionStateTerminated = 2; } ``` Instead of having fixed fields for specifying values such as required session viewers and roles this model centers around conditional allow rules and filters. It is implemented as a bi-directional mapping between the role of the session initiator and the roles of the other session participants. Roles can have a `requiresessionjoin` rule under `allow` containing requirements for session participants before a session may be started with privileged access to nodes that the role provides. Roles can also have a `join_sessions` rule under `allow` specifying which roles and session types that the role grants privileges to join. We will only initially support the modes `moderator` for Kubernetes Access and `peer` for SSH sessions. An `observer` mode also exists which only grants access to view but does not terminate an ongoing session. This RBAC model replaces the existing RBAC model for accessing SSH sessions. The existing model allows you to join all sessions to a node that you have login access to. If this is kept, this new RBAC model becomes inflexible as it is no longer possible to configure observers or moderators that do not themselves have access to start a session. The practical implication of this is that we no longer perform RBAC authorization at the node level when joining sessions, but instead deferring all authorization duties to the downstream authorizer for the session. Imagine you have 4 roles: `prod-access` `senior-dev` `customer-db-maintenance` `maintenance-observer` And these requirements: `prod-access` should be able to start sessions of any type with either one `senior-dev` participant two `dev` participants. `senior-dev` should be able to start sessions of any type without oversight. `customer-db-maintenance` needs oversight by one `maintenance-observer` on `ssh` type sessions. Then the 4 roles could be defined as follows: ```yaml kind: role metadata: name: prod-access spec: allow: requiresessionjoin: name: Senior dev oversight filter: 'contains(observer.roles,"senior-dev")' kinds: ['k8s', 'ssh'] modes: ['moderator'] count: 1 name: Dual dev oversight filter: 'contains(observer.roles,"dev")' kinds: ['k8s', 'ssh'] modes: ['moderator'] ``` ```yaml kind: role metadata: name: senior-dev spec: allow: join_sessions: name: Senior dev oversight roles : ['prod-access', 'training'] kinds: ['k8s', 'ssh', 'db'] modes: ['moderator'] ``` ```yaml kind: role metadata: name: customer-db-maintenance spec: allow: requiresessionjoin: name: Maintenance oversight filter: 'contains(observer.roles, "maintenance-observer")' kinds: ['ssh'] modes: ['moderator'] count: 1 ``` ```yaml kind: role metadata: name: maintenance-observer spec: allow: join_sessions: name: Maintenance oversight roles: ['customer-db-*'] kind: ['*'] modes: ['moderator'] ``` A filter determines if a user may act as an approved observer or not. To facilitate more complex configurations which may be desired we borrow some ideas from the `where` clause used by resource rules. To make it more workable, the language has been slimmed down significantly to handle this particular use case very well. `set["key"]`: Set and array indexing `contains(set, item)`: Determines if the set contains the item or not. `user` ```json { "traits": "map<string, []string>", "roles": "[]string", "name": "string" } ``` The grammar and other language is otherwise equal to that of the `where` clauses used by resource rules and the language used by approval requests, This promotes consistency across the product, reducing