Project: Teleport (Provisioning / Security & Compliance)
File: 0073-public-image-registry.md

authors: Logan Davis (logan.davis@goteleport.com) state: implemented Teleport images are currently hosted on . This RFD proposes migrating public images from Quay to . As of August 1st, 2021 Quay.io no longer supports any other authentication provider other than Red Hat Single-Sign On. Users in the Gravitational organization on Quay must be manually removed when they leave Teleport which presents a potential security risk. Migrating to Amazon ECR will consolidate our infrastructure while also improving security with support for IAM policies and our existing SSO infrastructure. Teleport will use Amazon ECR and Amazon ECR Public in order to host public container images. The multiple stage registry pipeline will allow Teleport to test and verify images internally before promoting them to our customers. As of authoring this RFD, Amazon ECR Public lacks support for . The two-stage pipeline will allow us to leverage these features in the internal repository before pushing to the public. What about name squatting on other container registry platforms? A not yet finished RFD on third-party artifact mirroring will address concerns regarding name squatting on other container registry platforms. See . This RFD will focus on the infrastructure, security, and observability of the replacement registry. It will also include a deprecation and migration plan for the existing Quay.io repositories. Infrastructure plans with Migration and Deprecation plan Image Signing w/ Cosign Mirroring of images to third-party registries (other than Quay.io). See . ``` Tag / Push Promotion AWS ECR AWS ECR Public public.ecr.aws/gravitational/teleport AWS Account: teleport-prod Quay.io quay.io/gravitational/teleport ``` The infrastructure will live in the repository. The terraform for the `teleport-prod` account can be found . Using AWS ECR and ECR Public allow us to rely on their managed infrastructure which reduces the operational complexity while enforcing our own security policies and allowing us to better audit changes to the environment. For more information on the pros and cons of alternatives, see . Amazon ECR Public assigns a random, default registry alias when the first public repository is created. However, a custom alias, such as `teleport` or `gravitational` can be requested. This alias is the display name in the Amazon ECR Public Gallery. Since the recent chose to use a convention of `gravitational/teleport`. I propose we request the `gravitational` alias in order to stay consistent amongst our artifacts. Thus, the image would be available to pull from `public.ecr.aws/gravitational/teleport`. Most of the security standards that will be applied to the infrastructure for this RFD are defined in Cloud RFD 17 - . All employee interaction with the registry and repositories will require our existing Okta SSO w/ MFA. Teleport employees will have read access to the internal ECR registry in order to test images before promoting them. Teleport employees will not have direct write access to

Teleport Release Engineers responsible for the artifacts will have limited write access through an assumed role. This role must be logged to the audit logs. Release engineers shall have no ability to modify or change audit logs. In case of an urgent fix, core release engineers will be able to assume an admin role that gives them permissions to write to repositories. This role assumption and all other actions taken by the engineer will be audited and alerted upon as defined in the . Service Accounts with least privilege permissions will handle pushing and promoting the images to the registries. For instance, a service account that is used during tags will have limited access to push to the internal ECR repositories. Another service account that handles promotions will have access to pull from AWS ECR and push to AWS ECR Public. Example terraform for the repository and promotion can be seen Amazon ECR provides detailed usage metrics through as well as detailed logging through AWS . For Amazon ECR Public, observability is limited. Currently, you can log authenticated requests via . An open issue exists for better metrics for ECR Public, see . Using the list of public teleport images defined , the terraform infrastructure needed for these registries will be created according to the standards defined in. Existing CI/CD pipelines and tooling will be updated to push to both Quay.io and ECR. Documentation will be updated to reflect the new registry location. Quay.io images will be deprecated and removed in reverse level order. (Bronze -> Silver -> Gold) Gold standard images (teleport, teleport-ent, etc...) will continue to exist and be pushed to Quay.io for the foreseeable future. will go into more details. The registry is an open source implementation of the specification. is an OSS, all-in-one, registry solution that is built on top of the docker registry. Harbor has a built-in UI, support for OIDC authentication and many more . While Harbor provides a maximally featured container registry solution, it also incurs an increased operational overhead that Teleport didn't have with Quay.io. A minimal, oci-compatible registry could be implemented through just CloudFront functions. This registry would only support reading. This would reduce the operational complexity of the current strategy to AWS specific components. Additional components would be needed to be developed in order to push the image to the S3 bucket but could be implemented as just another step in the existing CI/CD pipeline. A negative to this solution is the lack of features that come standard with other registry solutions. This includes, but is not limited to, vulnerability scanning and tag immutability. Additionally, discoverability would be a missing feature from this solution. and are Google's container registry solution where the former is a subset of the latters capabilities. They support for a , but lack tag

With support for multi-regional deployment, GCR / Artifact Registry provide a similar experience for both the Teleport team and our customers. However, due to Teleports stronger operational experience using AWS, we recommend ECR over GCR / Artifact Registry. ```hcl resource "awsecrrepository" "teleport" { name = "gravitational/teleport" imagetagmutability = "IMMUTABLE" encryption_configuration { encryption_type = "KMS" kmskey = awskmskey.ecrkey.arn } imagescanningconfiguration { scanonpush = true } } resource "awsecrpublicrepository" "teleport" { repository_name = "teleport" catalog_data { ... } } data "awsiampolicydocument" "teleportpromotion_user" { statement { sid = "AllowPushImages" effect = "Allow" actions = [ "ecr:BatchCheckLayerAvailability", "ecr:CompleteLayerUpload", "ecr:InitiateLayerUpload", "ecr:PutImage", "ecr:UploadLayerPart" ] resources = [awsecrpublicrepository.teleport.arn] } statement { sid = "AllowPullImages" effect = "Allow" actions = [ "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", ] resources = [awsecrrepository.teleport.arn] } statement { sid = "AllowAuthToken" effect = "Allow" actions = [ "ecr:GetAuthorizationToken", ] resources = ["*"] } } resource "awsiampolicy" "teleportpromotionuser" { name = "teleportpromotionuser" path = "/" description = "Amazon ECR Policy for promoting teleport from internal ECR to public ECR." policy = data.awsiampolicydocument.teleportpromotion_user } resource "awsiamuser" "teleportpromotionuser" { name = "teleportpromotionuser" } resource "awsiamuserpolicyattachment" "teleportpromotionuser" { user = awsiamuser.teleportpromotionuser.name policyarn = awsiampolicy.teleportpromotion_user.arn } ``` The following table represents a best guess guide to migration of existing images from Quay to Harbor. They have been marked as such given their perceived relevance based on Quay activity and number of references in the Gravitational organization. | Repositories | | - | | teleport | | teleport-ent | | teleport-plugin-email | | teleport-plugin-event-handler | | teleport-plugin-jira | | teleport-plugin-mattermost | | teleport-plugin-pagerduty | | teleport-plugin-slack | | Repositories | | - | | aws-ecr-helper | | buildbox-base | | fpm-centos | | fpm-debian | | next | | prometheus-operator | | teleport-buildbox | | teleport-buildbox-arm | | teleport-buildbox-arm-fips | | teleport-buildbox-fips | | teleport-ent-dev | | teleport-operator | | Repositories | | - | | debian-grande | | debian-tall | | debian-venti | | kube-router | | gravity-scan | | pithos | | pithosctl | | pithos-proxy | | planet | | robotest | | robotest-e2e | | robotest-suite | | rig | | satellite | | stolon | | stolonctl | | stolon-etcd | | stolon-pgbouncer | | teleport-lab | | ubuntu-grande | | ubuntu-tall | | ubuntu-venti | | wormhole | | wormhole-dev | | Repositories | | - | | alpine | | alpine-glibc | | busyloop | | cve-2018-1002105 | | docker-alpine-build | | docker-gc | | drone-fork-approval-extension | | force | | kaniko-init-container | | keygen | | mattermost-worker | | mkdocs-base | | monitoring-grafana | | monitoring-influxdb | | netbox | | nethealth-dev | | nginx | | nginx-controller | | provisioner | | s3-mounter | | slackbot | | stress | | sync-kubeconfig | | sync-kubesecrets | | teleport-buildbox-centos6 | | teleport-buildbox-centos6-fips | | teleport-dev | | tube | | watcher |