Project: Teleport (Provisioning / Security & Compliance)
File: 0041-aws-node-join.md

authors: Nic Klaassen (nic@goteleport.com) state: implemented Teleport nodes running on EC2 instances will be able to join a teleport cluster without the need to share any secret token with the node. Instead, the node will provide either a signed or a signed request that will be used to confirm the AWS account that the EC2 instance is running in. With a configured AWS join token on the auth server, the node will be allowed to join the cluster based on its AWS account. This will make provisioning nodes on EC2 simpler and arguably more secure. There is no need to configure static tokens, or to manage the provisioning and deployment of dynamic tokens. We are currently exploring two ways to implement this, both inspired by [similar designs](https://www.vaultproject.io/docs/auth/aws) used by Vault. The first is the "EC2 Method", which uses signed EC2 Instance Identity Documents which EC2 instances can fetch from the AWS . The second is the "IAM Method", in which the EC2 instance will create and sign an HTTPS API request to Amazon's public `sts:GetCallerIdentity` endpoint and pass it to the auth server. Update 2022-01-24: The EC2 Method was implemented first and released in Teleport 7.3, the IAM method is currently being implemented and will support Teleport Cloud. In place of a join token, nodes will present an EC2 IID (Instance Identity Document) to the auth server, with a corresponding PKCS7 signature. The IID and signature will be fetched on the nodes from the IMDSv2. This will be sent along with the other existing parameters in a [RegisterUsingTokenRequest]( https://github.com/gravitational/teleport/blob/d4247cb150d720be97521347b74bf9c526ae869f/lib/auth/auth.go#L1538-L1563). The auth server will then: Check that the `PendingTime` of the IID, which is the time the instance was launched, is within the configured TTL. if the node fails to join the cluster during this window, the user can stop and restart the EC2 instance to reset the `PendingTime` (and effectively create a new IID with a new signature) Check that the AWS join token matches the AWS `account` and `region` in the IID, and the requested Teleport service role. Use the AWS DSA public certificate to check that the PKCS7 signature for the IID is valid. The `region` field of the IID will be used to select the correct public certificate (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/verify-pkcs7.html). Check that this EC2 instance has not already joined the cluster. The node name will be set to `<awsaccountid>-<awsinstanceid>` so that the auth server can efficiently check the backend to see whether this instance has already joined. This is to prevent replay attacks with the IID, any attempt by the same instance to join the cluster more than once will be blocked and logged in detail. Check that the EC2 instance is currently running. \* \* Step 5 requires AWS credentials on the Auth server with permissions for

If you need Nodes to be able to join the teleport cluster from different AWS accounts than the one in which the Teleport Auth server is running, there are a few extra required steps: Every AWS account in which Nodes should be able to join will need an IAM role which can be assumed from the account of the Auth server. This role needs permissions for `ec2:DescribeInstances`. The IAM role of the Auth server needs permissions to `sts:AssumeRole` for the ARN of every foreign IAM role from step 1. These ARNs will need to be configured in the `aws_role` field of the AWS join token. The design described above is not ideal for Teleport Cloud, as the auth server would need permissions to access customer AWS accounts to check that the instance is currently running. To make this work, we could create a new type of teleport agent that the client would run on their own infrastructure, similar to app and db access. The customer would need to configure the `ec2:DescribeInstances` permissions for that agent only. In order to check that the joining instance is currently running, the Auth server would delegate the `ec2:DescribeInstances` call to the agent running in the customer's account. In place of a join token, nodes will present a signed `sts:GetCallerIdentity` request to the auth server. The form of this request is an HTTP `POST` request to `sts.amazonaws.com` with `Action=GetCallerIdentity` in the body. The request can include arbitrary headers which will also be signed using the . The signature will then be included in the `Authorization` header of the request. See Appendix I for an example. The actual creation and signing of the request will be done on the node with the AWS SDK for Go. Because the signed request can include arbitrary headers, this allows us to issue a challenge (a crypto random string of bytes) that the node must include in the signed request. We can make use of gRPC bidirectional streams to implement the following flow at a new `RegisterWithAWSToken` gRPC endpoint: The node initiates a `RegisterWithAWSToken` request. The auth server enforces that TLS is used for transport. The auth server sends a base64 encoded 32 byte crypto-random challenge. The node creates the `sts:GetCallerIdentity` request, adds the challenge in a header, and signs it with its AWS credentials. The node sends a join request to the auth server, including the signed `sts:GetCallerIdentity` request and other parameters currently in [RegisterUsingTokenRequest]( https://github.com/gravitational/teleport/blob/d4247cb150d720be97521347b74bf9c526ae869f/lib/auth/auth.go#L1538-L1563) The auth server checks that the signed request is valid: it is a `POST` request to `sts.amazonaws.com` the body is `Action=GetCallerIdentity&Version=2011-06-15` the `X-Teleport-Challenge` header value equals the issued challenge the Authorization header includes `x-teleport-challenge` as one of the signed headers the Authorization header includes `AWS4-HMAC-SHA256` as the signing algorithm The auth server sends the signed request to `sts.amazonaws.com` over TLS and gets the response. The auth server checks for a provision token that matches the AWS account in the `sts:GetCallerIdentity` response and the teleport service role of the join request. The auth server sends credentials to join the

The above steps all occur within a single streaming gRPC request and only 1 attempt with a given challenge will be allowed. The complete request will have a 1 minute timeout. In order to create the signed requests, the node will need to have access to AWS credentials. As the `sts:GetCallerIdentity` request does not require any explicit permissions, it is sufficient to attach any IAM policy to the EC2 instance. Note that cross-account `sts:AssumeRole` access should be restricted for all IAM roles in the AWS account to prevent possible attackers in other accounts from pretending they are in the configured AWS account. To mitigate this, administrators can restrict access to only a (set of) IAM roles(s) which cannot be assumed from other accounts, see #Teleport Configuration. | | EC2 | IAM | Dynamic Token | | | | | - | | Requires a secret token to be shared with every Node | | | | | Requires attached IAM role on Nodes | | | | | Requires attached IAM role on Auth | | | | Requires IAM role in every Node account which can be assumed by Auth account | | | | | Requires IAM role in Auth account which can assume roles in every Node account | | | | | Could work with Teleport Cloud | \* | | | | Nodes can sign a challenge to prevent replays | | | | | Requires outbound access to public internet from Auth | | | | | Requires outbound access to public internet from Nodes | | | | The existing provision token type can be extended to support aws authentication. Token "rules" will be added to define which AWS accounts and IAM roles to accept nodes from. Support for `tctl create token.yaml`, `tctl get tokens`, and `tctl rm tokens/example_token` will need to be added so that administrators can create and modify tokens from a yaml definition. ```yaml kind: token version: v2 metadata: name: exampleawstoken spec: roles: [Node,Kube,Db] allow: aws_account: "111111111111" aws_role: "arn:aws:iam::111111111111:role/example-role" aws_regions: ["us-west-2", "us-east-1"] aws_account: "222222222222" aws_role: "arn:aws:iam::222222222222:role/example-role" aws_regions: ["us-gov-east-1"] aws_account: "333333333333" aws_role: "arn:aws:iam::333333333333:role/other-example-role" awsiidttl: 5m ``` teleport.yaml on the nodes should be configured so that they will use the new aws join token: ```yaml teleport: join_params: tokenname: "exampleaws_token" # should match the name of the token on the auth server method: ec2 ``` `sts:GetCallerIdentity` Request ``` POST / HTTP/1.1 Host: sts.amazonaws.com Accept: application/json Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXX/20210614/us-east-1/sts/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date, Signature=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Content-Length: 43 Content-Type: application/x-www-form-urlencoded; charset=utf-8 X-Amz-Date: 20210614T014047Z Action=GetCallerIdentity&Version=2011-06-15 ``` Response: ``` { "GetCallerIdentityResponse": "GetCallerIdentityResult": { "Account":"111111111111", "Arn":"arn:aws:iam::111111111111:assumed-role/test-role/i-RRRRRRRRRRRRRRRRRR", "UserId":"AAAAAAAAAAAAAAAAAAAAA:i-RRRRRRRRRRRRRRRRRR" }, "ResponseMetadata":{ "RequestId":"4464f2b3-36ba-4dd5-b0a7-e9c4fbd7b568" } } } ``` ```json { "accountId" : "111111111111", "architecture" : "x86_64", "availabilityZone" : "us-west-2a", "billingProducts" : null, "devpayProductCodes" : null, "marketplaceProductCodes" : null, "imageId" : "ami-00000000000000000", "instanceId" : "i-01111111111111111", "instanceType" : "t2.micro", "kernelId" : null, "pendingTime" : "2021-06-11T00:08:27Z", "privateIp" : "10.0.0.56", "ramdiskId" : null, "region" : "us-west-2", "version" : "2017-09-30" } ``` The pkcs7 signature is separate from the document and looks like: ``` MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEggHZewog ICJhY2NvdW50SWQiIDogIjI3ODU3NjIyMDQ1MyIsCiAgImFyY2hpdGVjdHVyZSIgOiAieDg2XzY0 IiwKICAiYXZhaWxhYmlsaXR5Wm9uZSIgOiAidXMtd2VzdC0yYSIsCiAgImJpbGxpbmdQcm9kdWN0 cyIgOiBudWxsLAogICJkZXZwYXlQcm9kdWN0Q29kZXMiIDogbnVsbCwKICAibWFya2V0cGxhY2VQ cm9kdWN0Q29kZXMiIDogbnVsbCwKICAiaW1hZ2VJZCIgOiAiYW1pLTA4MDBmYzBmYTcxNWZkY2Zl IiwKICAiaW5zdGFuY2VJZCIgOiAiaS0wMjg1Yjc2ZGJjOGY3NWNlNiIsCiAgImluc3RhbmNlVHlw ZSIgOiAidDIubWljcm8iLAogICJrZXJuZWxJZCIgOiBudWxsLAogICJwZW5kaW5nVGltZSIgOiAi MjAyMS0wNi0xMVQwMDowODoyN1oiLAogICJwcml2YXRlSXAiIDogIjEwLjAuMC41NiIsCiAgInJh bWRpc2tJZCIgOiBudWxsLAogICJyZWdpb24iIDogInVzLXdlc3QtMiIsCiAgInZlcnNpb24iIDog IjIwMTctMDktMzAiCn0AAAAAAAAxggEXMIIBEwIBATBpMFwxCzAJBgNVBAYTAlVTMRkwFwYDVQQI ExBXYXNoaW5ndG9uIFN0YXRlMRAwDgYDVQQHEwdTZWF0dGxlMSAwHgYDVQQKExdBbWF6b24gV2Vi IFNlcnZpY2VzIExMQwIJAJa6SNnlXhpnMAkGBSsOAwIaBQCgXTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMTA2MTEwMDA4MzBaMCMGCSqGSIb3DQEJBDEWBBSj4yyW kvOVpx656w8wuhjUyH9dWjAJBgcqhkjOOAQDBC4wLAIUBOSAnp6M57kAXXJoj2s8cb32AXwCFFHm egSjvG+KBmmOgUyS15ZzFJJTAAAAAAAA ```