Project: Teleport (Provisioning / Security & Compliance)
File: 0094-kubernetes-node-joining.md

authors: Hugo Hervieux (hugo.hervieux@goteleport.com) state: implemented Engineering: @r0mant && @gus Product: @klizhentas && @xinding33 Security: @reedloden This RFD discusses various methods for Teleport proxies to join a Teleport cluster in a same-cluster Kubernetes context and covers implementation details of the chosen approach. We want to refactor the `teleport-cluster` Helm chart to be a reference delivery method for Teleport. This involves deploying separately Teleport auth and proxy nodes. Proxy nodes have to register against the Teleport auth nodes without user interaction and without compromising the cluster security (no long-lived tokens). Proxy nodes can be automatically created or deleted (by a Kubernetes HPA for example). User Motivation: As new Teleport adopter, I want to deploy a working, secure and highly available Teleport cluster on Kubernetes, with a single Helm command and as few values as possible so that I can start using Teleport as fast as possible. As a Teleport power user, I want to easily join my other co-hosted Teleport services (app, db, discovery, windows, Machine ID, ...) on Kubernetes so that I have less join automation to build and maintain. The first section of this RFD explores three different approaches on how to have proxies automatically join the cluster. The next sections discuss implementation details and security/ux considerations of the chosen approach. In this approach we introduce a new `kube` join method relying on Kubernetes tokens to establish trust (like the `ec2` and `iam` ones). Teleport tokens using this method can be long-lived with less security risk because they are not secret and rely on Kubernetes as a trusted third party. ```mermaid graph LR helmChart serviceAccount token deployment pod cluster kubernetes helmChart --> serviceAccount helmChart --> deployment helmChart --> |create a new Kubernetes join token| cluster serviceAccount --> |on-demand| token token --> |mounted and bound to| pod deployment --> pod pod --> |join with SA token| cluster cluster --> |check SA token| kubernetes ``` This approach is basically extending . This brings the following challenges: helmChart has to create a Teleport join token for its service account: this can be done through the Teleport operator and a TeleportToken resource (we are due to implement tokens anyway to reach parity between the operator and the Terraform provider) this can be done through a bootstrap resource on cluster creation Kubernetes only supports (read enabled by default) short-lived renewed token after 1.20 (it depends on the Kubernetes distribution, but starting with 1.20 everyone must support it). For prior versions, users might have to use the legacy non-bound tokens produced by Kubernetes. In this approach, proxy certificates are delivered by

```mermaid flowchart LR cluster[Teleport cluster] tBot stateA podA stateB podB subgraph statefulset/deployment podA podB end cluster <-->|part of cluster| tBot tBot --> stateA tBot --> stateB stateA |uses| podA stateB |uses| podB ``` This approach brings the following challenges: each proxy requires multiple certs (one with the default teleport node role, the other with the proxy role) teleport does not support joining from certs (unless we manually build a valid state) `tbot` has to dynamically discover how many pods want certs and which certs they need (we don't have a streaming destination API yet) pods should have a way to access their specific state: we can't mount a different secret in each pod it can be done through admission webhooks pods can read secrets (this works OK for Teleport nodes, but cannot be generalized with other workloads like access plugin because this requires being able to read all certs from the namespace) pods can request a cert to tBot over HTTP `tbot` has to validate the identity of the pod requesting the certificate `tbot` has to join the cluster initially, which is almost the same problem that we're trying to address (almost because we have less constraints due to autoscaling). The easiest approach would be to bundle `tbot` with auth nodes and use the local admin socket to bootstrap it, like what we're doing with the operator In this approach, short-lived static tokens are generated by the Teleport operator and injected in containers using mutation webhooks. ```mermaid sequenceDiagram StatefulSet->>Kubernetes: New pod Kubernetes->>+Operator: Mutation webhook Operator->>Teleport Cluster:New token, ttl = 2min Teleport Cluster->>Operator: Token Operator->>Kubernetes: Create secret with token Operator->>-Kubernetes: Mutated pod with secret mount Kubernetes->>Proxy: Start Note over Operator: Wait for token expiry Operator->>Kubernetes: Delete secret ``` The operator internally relies on MachineID to get its identity, this solution is basically an automated version of [MachineID token generation](https://github.com/gravitational/teleport/discussions/15404). This approach brings the following challenges: the operator becomes mandatory, which might not be a possibility for certain users (because of new additional RBAC) the operator enters the critical path, if it is not available no pod can start pods started before the mutation webhook is added will not get their token there is a race condition if the proxy pod does not start before expiry, the failure mode is a pod stuck in crashloop and the only way out is to delete the pod. This is particularly likely to happen on clusters with node autoscaling or capacity issues. The second approach tends to bring more issues than it solves. Even if we added MachineID support for Teleport nodes, this would not be used by customers with their Teleport workload. The third approach adds new components in the critical path and new complex failure modes. The first approach seems the more robust, leverages previous work with ProvisionTokens and provides very solid security guarantees for Kubernetes 1.20+. MachineID would also benefit from it as `tBot` would be easily deployed in a Kubernetes cluster hosting a Teleport cluster. Note: the term `token` is confusing in the context of this RFD as it covers both Kubernetes-issued tokens and Teleport join tokens of type

In the following doc, "Kubernetes token" and "service account token" refer to a Kubernetes-issued token and "join token" refers to a Teleport join token of type `kube`. If this RFD is accepted, the teleport documentation should reflect this nuance and using the term token alone should be avoided. The feasibility of this approach has been tested against Multiple Kubernetes setups, including: Kubernetes 1.18 on Minikube Kubernetes 1.20 on EKS Kubernetes 1.22 on GKE The proof of concept code can be found . A teleport cluster is running in Kubernetes. An operator (human or automated) wants to deploy a new set of Teleport proxies in the same cluster. The operator creates a new `kube` join token, allowing the kubernetes service account `new-proxies` The operator creates a Kubernetes service account `new-proxies` The operator creates its new pods (with a Deployment, a StatefulSet, or even a DaemonSet) using the `new-proxies` SA The Teleport proxies uses the mounted Kubernetes token to join the existing Teleport cluster The Teleport cluster validates the token against Kubernetes using the `TokenReview` API The proxies are part of the cluster All Kubernetes tokens contain the service account name and its groups. Versions 1.20 and above ([bound tokens](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md#boundserviceaccounttokenvolume-1)): tokens are signed per pod tokens are automatically renewed by the kubelet token TTL is configurable in the `PodSpec` tokens are revoked when the pod is deleted Versions prior 1.20: tokens are signed per service-account tokens are long-lived tokens are revoked when the service account is deleted Tokens are JWTs, Kubernetes JWKS can be manually fetched and JWT validation can be done offline. Kubernetes also exposes a to validate tokens. This API performs more checks, like revocation, UUID checking pod and service account, ... This API is the sanest way to validate Kubernetes token. The teleport resource would look like: ```yaml kind: token version: v2 metadata: name: proxy-token expires: "3000-01-01T00:00:00Z" spec: roles: ["Proxy"] k8s: allow: service_account: "my-namespace:my-service-account" ``` The `allow[*].service_account`check should match exactly the `User` field of the Kubernetes token stripped of the `system:serviceaccounts:` prefix. Checking if the `system:serviceaccounts` prefix is present ensures no Kubernetes user tokens can be used to join the Teleport cluster. Supporting this token kind does not require changes to the `RegisterUsingTokenRequest` type, the Kubernetes token should be passed through the `IDToken` field and the Teleport token name should be passed in the `Token` field. The join logic would be similar to what is done with GitHub joining. Teleport should validate on token creation if it can access the `TokenReview` API by reviewing its own Kubernetes token. This will allow to fail fast on broken or not supported setups. The token spec could be extended in a second phase to support trusting other Kubernetes apiservers than the one where the auth is running. This will imply adding fields to the ProvisionToken to specify how Teleport should authenticate to Kubernetes. As a large portion of clusters are hosted by cloud providers, we will likely need to support the biggest cloud authentication

The token would look like this: ```yaml kind: token version: v2 metadata: name: proxy-token expires: "3000-01-01T00:00:00Z" spec: roles: ["Proxy"] k8s: allow: service_account: "my-namespace:my-service-account" authType: "inCluster|kubeconfig|gcp|aws|azure|rancher|..." kubeconfig: certificate-authority-data: base64 server: http://1.2.3.4 token: my-secret-token user: client-sa-name client-certificate-data: base64 client-key-data: base64 gcp: service-account-key: base64 service-account-name: "foo@sa-project.iam.gserviceaccount.com" project: cluster-project location: us-east1 cluster-name: my-cluster aws: key-id: ASDFGHJK key-secret: QWERTYUIOP region: us-east-1 account-id: 1234567890 cluster-name: my-cluster azure: rancher: ``` For cloud providers like GCP or AWS, users will have to make sure the cloud service-account is mapped to a Kubernetes group bound to a ClusterRole allowing to use the `TokenReview` API. As this setup can be complex, Teleport should validate on token creation it has access to the `TokenReview` API. It can do so by reviewing its own Kubernetes token. Bound tokens mounted through projected volumes offer a solid level of security as those tokens are short-lived, automatically renewed and are revoked on the pod deletion. Those tokens are the pod's identity from a Kubernetes point of view. Legacy tokens offer less security guarantees as they are long-lived and not automatically rotated. Joining using them is still an improvement over static tokens because the Kubernetes-issued token doesn't need to leave the cluster by design: human operators and the CI/CD system should never be in contact with the token. Legacy tokens are mostly a user experience improvement rather than a security improvement. The Teleport Helm charts should always favor using bound tokens. Kubernetes support for bound tokens must be detected at the deploy time and the chart must adapt its behaviour to maximize security based on Kubernetes capabilities. To mitigate token stealing or replaying MiTM attack, the joining Teleport nodes should be configured with the ca_pins, as for regular tokens. Introducing the feature does not add new critical components that could cause availability issues. Creating new pods already requires a healthy apiserver. If the apiserver were to fail, this would not affect already joined proxies. While joining Teleport proxy nodes is the main motivation of this proposal, other Teleport nodes would also benefit from this additional join method. Deploying nodes in the same Kubernetes cluster will become easier. This can be used to join all kinds of Teleport service (kubernetes, app, db, windows desktop, discovery), but also to join `tbot` and offer an easier Machine ID integration in Kubernetes. This would pave the way for a better experience for users hosting access plugins next to the Teleport control plane. ```mermaid graph LR subgraph cluster helm chart cluster tBot subgraph accessPlugin chart accessPlugin end cluster |joined using the 'kube' method| tBot tBot --> |store certs| secret secret |mounted| accessPlugin accessPlugin --> |connects with its MachineID certs| cluster end ``` Joining nodes from outside the Kubernetes cluster would imply generating and exporting a Kubernetes token from the service account. While this pattern is feasible, it does not provide much added value compared to generating a static token from Teleport directly.