Chapter 3. Customer Cloud Subscriptions on AWS OpenShift Dedicated provides a Customer Cloud Subscription (CCS) model that allows Red Hat to deploy and manage clusters into a customer's existing Amazon Web Service (AWS) account. 3.1. Understanding Customer Cloud Subscriptions on AWS To deploy OpenShift Dedicated into your existing Amazon Web Services (AWS) account using the Customer Cloud Subscription (CCS) model, Red Hat requires several prerequisites be met. Red Hat recommends the usage of an AWS Organization to manage multiple AWS accounts. The AWS Organization, managed by the customer, hosts multiple AWS accounts. There is a root account in the organization that all accounts will refer to in the account hierarchy. It is recommended for the OpenShift Dedicated cluster using a CCS model to be hosted in an AWS account within an AWS Organizational Unit. A service control policy (SCP) is created and applied to the AWS Organizational Unit that manages what services the AWS sub-accounts are permitted to access. The SCP applies only to available permissions within a single AWS account for all AWS sub-accounts within the Organizational Unit. It is also possible to apply a SCP to a single AWS account. All other accounts in the customer's AWS Organization are managed in whatever manner the customer requires. Red Hat Site Reliability Engineers (SRE) will not have any control over SCPs within the AWS Organization. 3.2. Customer requirements OpenShift Dedicated clusters using a Customer Cloud Subscription (CCS) model on Amazon Web Services (AWS) must meet several prerequisites before they can be deployed. 3.2.1. Account The customer ensures that AWS limits are sufficient to support OpenShift Dedicated provisioned within the customer-provided AWS account. The customer-provided AWS account should be in the customer's AWS Organization with the applicable service control policy (SCP) applied. Note It is not a requirement that the customer-provided account be within an AWS Organization or for the SCP to be applied, however Red Hat must be able to perform all the actions listed in the SCP without restriction. The customer-provided AWS account must not be transferable to Red Hat. The customer may not impose AWS usage restrictions on Red Hat activities. Imposing restrictions severely hinders Red Hat's ability to respond to incidents. Red Hat deploys monitoring into AWS to alert Red Hat when a highly privileged account, such as a root account, logs into the customer-provided AWS account. The customer can deploy native AWS services within the same customer-provided AWS account. Note Customers are encouraged, but not mandated, to deploy resources in a Virtual Private Cloud (VPC) separate from the VPC hosting OpenShift Dedicated and other Red Hat supported services. 3.2.2. Access requirements To appropriately manage the OpenShift Dedicated service, Red Hat must have the AdministratorAccess policy applied to the administrator role at all times. Note This policy only provides Red Hat with permissions and capabilities to change resources in the customer-provided AWS account. Red Hat must have AWS console access to the customer-provided AWS account. This access is protected and managed by Red Hat. The customer must not utilize the AWS account to elevate their permissions within the OpenShift Dedicated cluster. Actions available in OpenShift Cluster Manager must not be directly performed in the customer-provided AWS account. 3.2.3. Support requirements Red Hat recommends that the customer have at least Business Support from AWS. Red Hat has authority from the customer to request AWS support on their behalf. Red Hat has authority from the customer to request AWS resource limit increases on the customer-provided account. Red Hat manages the restrictions, limitations, expectations, and defaults for all OpenShift Dedicated clusters in the same manner, unless otherwise specified in this requirements section. 3.2.4. Security requirements The customer-provided IAM credentials must be unique to the customer-provided AWS account and must not be stored anywhere in the customer-provided AWS account. Volume snapshots will remain within the customer-provided AWS account and customer-specified region. Red Hat must have ingress access to EC2 hosts and the API server through white-listed Red Hat machines. Red Hat must have egress allowed to forward system and audit logs to a Red Hat managed central logging stack. 3.3. Required customer procedure The Customer Cloud Subscription (CCS) model allows Red Hat to deploy and manage OpenShift Dedicated into a customer's Amazon Web Services (AWS) account. Red Hat requires several prerequisites in order to provide these services. Procedure If the customer is using AWS Organizations, you must either use an AWS account within your organization or create a new one . To ensure that Red Hat can perform necessary actions, you must either create a service control policy (SCP) or ensure that none is applied to the AWS account. Attach the SCP to the AWS account. Within the AWS account, you must create an osdCcsAdmin IAM user with the following requirements: This user needs at least Programmatic access enabled. This user must have the AdministratorAccess policy attached to it. Provide the IAM user credentials to Red Hat. You must provide the access key ID and secret access key in OpenShift Cluster Manager . 3.4. Minimum required service control policy (SCP) Service control policy (SCP) management is the responsibility of the customer. These policies are maintained in the AWS Organization and control what services are available within the attached AWS accounts. Required/optional Service Actions Effect Required Amazon EC2 All Allow Amazon EC2 Auto Scaling All Allow Amazon S3 All Allow Identity And Access Management All Allow Elastic Load Balancing All Allow Elastic Load Balancing V2 All Allow Amazon CloudWatch All Allow Amazon CloudWatch Events All Allow Amazon CloudWatch Logs All Allow AWS Support All Allow AWS Key Management Service All Allow AWS Security Token Service All Allow AWS Resource Tagging All Allow AWS Route53 DNS All Allow AWS Service Quotas ListServices GetRequestedServiceQuotaChange GetServiceQuota RequestServiceQuotaIncrease ListServiceQuotas Allow Optional AWS Billing ViewAccount Viewbilling ViewUsage Allow AWS Cost and Usage Report All Allow AWS Cost Explorer Services All Allow { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "autoscaling:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "elasticloadbalancing:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "cloudwatch:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "events:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "logs:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "support:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "kms:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "sts:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "tag:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "route53:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "servicequotas:ListServices", "servicequotas:GetRequestedServiceQuotaChange", "servicequotas:GetServiceQuota", "servicequotas:RequestServiceQuotaIncrease", "servicequotas:ListServiceQuotas" ], "Resource": [ "*" ] } ] } 3.5. Red Hat managed IAM references for AWS Red Hat is responsible for creating and managing the following Amazon Web Services (AWS) resources: IAM policies, IAM users, and IAM roles. 3.5.1. IAM policies Note IAM policies are subject to modification as the capabilities of OpenShift Dedicated change. The AdministratorAccess policy is used by the administration role. This policy provides Red Hat the access necessary to administer the OpenShift Dedicated cluster in the customer-provided AWS account. { "Version": "2012-10-17", "Statement": [ { "Action": "*", "Resource": "*", "Effect": "Allow" } ] } The CustomerAdministratorAccess role provides the customer access to administer a subset of services within the AWS account. At this time, the following are allowed: VPC Peering VPN Setup Direct Connect (only available if granted through the service control policy) { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:AttachVpnGateway", "ec2:DescribeVpnConnections", "ec2:AcceptVpcPeeringConnection", "ec2:DeleteVpcPeeringConnection", "ec2:DescribeVpcPeeringConnections", "ec2:CreateVpnConnectionRoute", "ec2:RejectVpcPeeringConnection", "ec2:DetachVpnGateway", "ec2:DeleteVpnConnectionRoute", "ec2:DeleteVpnGateway", "ec2:DescribeVpcs", "ec2:CreateVpnGateway", "ec2:ModifyVpcPeeringConnectionOptions", "ec2:DeleteVpnConnection", "ec2:CreateVpcPeeringConnection", "ec2:DescribeVpnGateways", "ec2:CreateVpnConnection", "ec2:DescribeRouteTables", "ec2:CreateTags", "ec2:CreateRoute", "directconnect:*" ], "Resource": "*" } ] } If enabled, the BillingReadOnlyAccess role provides read-only access to view billing and usage information for the account. Billing and usage access is only granted if the root account in the AWS Organization has it enabled. This is an optional step the customer must perform to enable read-only billing and usage access and does not impact the creation of this profile and the role that uses it. If this role is not enabled, users will not see billing and usage information. See this tutorial on how to enable access to billing data . { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "aws-portal:ViewAccount", "aws-portal:ViewBilling" ], "Resource": "*" } ] } 3.5.2. IAM users The osdManagedAdmin user is created immediately after taking control of the customer-provided AWS account. This is the user that will perform the OpenShift Dedicated cluster installation. 3.5.3. IAM roles The network-mgmt role provides customer-federated administrative access to the AWS account through a separate AWS account. It also has the same access as a read-only role. The network-mgmt role only applies to non-Customer Cloud Subscription (CCS) clusters. The following policies are attached to the role: AmazonEC2ReadOnlyAccess CustomerAdministratorAccess The read-only role provides customer-federated read-only access to the AWS account through a separate AWS account. The following policies are attached to the role: AWSAccountUsageReportAccess AmazonEC2ReadOnlyAccess AmazonS3ReadOnlyAccess IAMReadOnlyAccess BillingReadOnlyAccess 3.6. Provisioned AWS Infrastructure This is an overview of the provisioned Amazon Web Services (AWS) components on a deployed OpenShift Dedicated cluster. For a more detailed listing of all provisioned AWS components, see the OpenShift Container Platform documentation . 3.6.1. AWS Elastic Computing (EC2) instances AWS EC2 instances are required to deploy the control plane and data plane functions of OpenShift Dedicated in the AWS public cloud. Instance types might vary for control plane and infrastructure nodes depending on worker node count. Single availability zone 3 m5.2xlarge minimum (control plane nodes) 2 r5.xlarge minimum (infrastructure nodes) 2 m5.xlarge minimum but highly variable (worker nodes) Multiple availability zones 3 m5.2xlarge minimum (control plane nodes) 3 r5.xlarge minimum (infrastructure nodes) 3 m5.xlarge minimum but highly variable (worker nodes) 3.6.2. AWS Elastic Block Store (EBS) storage Amazon EBS block storage is used for both local node storage and persistent volume storage. Volume requirements for each EC2 instance: Control plane volumes Size: 350 GB Type: io1 Input/output operations per second: 1000 Infrastructure volumes Size: 300 GB Type: gp2 Input/output operations per second: 900 Worker volumes Size: 300 GB Type: gp2 Input/output operations per second: 900 3.6.3. Elastic Load Balancing (ELB) load balancers Up to two Network Load Balancers for API and up to two Classic Load Balancers for application router. For more information, see the ELB documentation for AWS . 3.6.4. S3 storage The image registry and Elastic Block Store (EBS) volume snapshots are backed by AWS S3 storage. Pruning of resources is performed regularly to optimize S3 usage and cluster performance. Note Two buckets are required with a typical size of 2 TB each. 3.6.5. VPC Customers should expect to see one VPC per cluster. Additionally, the VPC needs the following configurations: Subnets : Two subnets for a cluster with a single availability zone, or six subnets for a cluster with multiple availability zones. Note A public subnet connects directly to the internet through an internet gateway. A private subnet connects to the internet through a network address translation (NAT) gateway. Route tables : One route table per private subnet, and one additional table per cluster. Internet gateways : One Internet Gateway per cluster. NAT gateways : One NAT Gateway per public subnet. 3.6.5.1. Sample VPC Architecture 3.6.6. Security groups AWS security groups provide security at the protocol and port-access level; they are associated with EC2 instances and Elastic Load Balancing. Each security group contains a set of rules that filter traffic coming in and out of an EC2 instance. You must ensure the ports required for the OpenShift Container Platform installation are open on your network and configured to allow access between hosts. 3.6.6.1. Additional custom security groups When you create a cluster by using a non-managed VPC, you can add custom security groups during cluster creation. Custom security groups are subject to the following limitations: You must create the custom security groups in AWS before you create the cluster. For more information, see Amazon EC2 security groups for Linux instances . You must associate the custom security groups with the VPC that the cluster will be installed into. Your custom security groups cannot be associated with another VPC. You might need to request additional quota for your VPC if you are adding additional custom security groups. For information on requesting an AWS quota increase, see Requesting a quota increase . 3.7. AWS firewall prerequisites If you are using a firewall to control egress traffic from OpenShift Dedicated, you must configure your firewall to grant access to the certain domain and port combinations below. OpenShift Dedicated requires this access to provide a fully managed OpenShift service. Prerequisites You have configured an Amazon S3 gateway endpoint in your AWS Virtual Private Cloud (VPC). This endpoint is required to complete requests from the cluster to the Amazon S3 service. Procedure Allowlist the following URLs that are used to install and download packages and tools: Domain Port Function registry.redhat.io 443 Provides core container images. quay.io 443 Provides core container images. cdn01.quay.io 443 Provides core container images. cdn02.quay.io 443 Provides core container images. cdn03.quay.io 443 Provides core container images. cdn04.quay.io 443 Provides core container images. cdn05.quay.io 443 Provides core container images. cdn06.quay.io 443 Provides core container images. sso.redhat.com 443 Required. The https://console.redhat.com/openshift site uses authentication from sso.redhat.com to download the pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, chargeback reporting, and so on. quay-registry.s3.amazonaws.com 443 Provides core container images. quayio-production-s3.s3.amazonaws.com 443 Provides core container images. registry.access.redhat.com 443 Hosts all the container images that are stored on the Red Hat Ecosytem Catalog. Additionally, the registry provides access to the odo CLI tool that helps developers build on OpenShift and Kubernetes. access.redhat.com 443 Required. Hosts a signature store that a container client requires for verifying images when pulling them from registry.access.redhat.com . registry.connect.redhat.com 443 Required for all third-party images and certified Operators. console.redhat.com 443 Required. Allows interactions between the cluster and OpenShift Console Manager to enable functionality, such as scheduling upgrades. sso.redhat.com 443 The https://console.redhat.com/openshift site uses authentication from sso.redhat.com . pull.q1w2.quay.rhcloud.com 443 Provides core container images as a fallback when quay.io is not available. catalog.redhat.com 443 The registry.access.redhat.com and https://registry.redhat.io sites redirect through catalog.redhat.com . oidc.op1.openshiftapps.com 443 Used by ROSA for STS implementation with managed OIDC configuration. Allowlist the following telemetry URLs: Domain Port Function cert-api.access.redhat.com 443 Required for telemetry. api.access.redhat.com 443 Required for telemetry. infogw.api.openshift.com 443 Required for telemetry. console.redhat.com 443 Required for telemetry and Red Hat Insights. observatorium-mst.api.openshift.com 443 Required for managed OpenShift-specific telemetry. observatorium.api.openshift.com 443 Required for managed OpenShift-specific telemetry. Managed clusters require enabling telemetry to allow Red Hat to react more quickly to problems, better support the customers, and better understand how product upgrades impact clusters. For more information about how remote health monitoring data is used by Red Hat, see About remote health monitoring in the Additional resources section. Allowlist the following Amazon Web Services (AWS) API URls: Domain Port Function .amazonaws.com 443 Required to access AWS services and resources. Alternatively, if you choose to not use a wildcard for Amazon Web Services (AWS) APIs, you must allowlist the following URLs: Domain Port Function ec2.amazonaws.com 443 Used to install and manage clusters in an AWS environment. events.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment. iam.amazonaws.com 443 Used to install and manage clusters in an AWS environment. route53.amazonaws.com 443 Used to install and manage clusters in an AWS environment. sts.amazonaws.com 443 Used to install and manage clusters in an AWS environment, for clusters configured to use the global endpoint for AWS STS. sts.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment, for clusters configured to use regionalized endpoints for AWS STS. See AWS STS regionalized endpoints for more information. tagging.us-east-1.amazonaws.com 443 Used to install and manage clusters in an AWS environment. This endpoint is always us-east-1, regardless of the region the cluster is deployed in. ec2.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment. elasticloadbalancing.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment. tagging.<aws_region>.amazonaws.com 443 Allows the assignment of metadata about AWS resources in the form of tags. Allowlist the following OpenShift URLs: Domain Port Function mirror.openshift.com 443 Used to access mirrored installation content and images. This site is also a source of release image signatures. api.openshift.com 443 Used to check if updates are available for the cluster. Allowlist the following site reliability engineering (SRE) and management URLs: Domain Port Function api.pagerduty.com 443 This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on. events.pagerduty.com 443 This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on. api.deadmanssnitch.com 443 Alerting service used by OpenShift Dedicated to send periodic pings that indicate whether the cluster is available and running. nosnch.in 443 Alerting service used by OpenShift Dedicated to send periodic pings that indicate whether the cluster is available and running. http-inputs-osdsecuritylogs.splunkcloud.com 443 Required. Used by the splunk-forwarder-operator as a logging forwarding endpoint to be used by Red Hat SRE for log-based alerting. sftp.access.redhat.com (Recommended) 22 The SFTP server used by must-gather-operator to upload diagnostic logs to help troubleshoot issues with the cluster. Additional resources About remote health monitoring 3.8. AWS account limits The OpenShift Dedicated cluster uses a number of Amazon Web Services (AWS) components, and the default service limits affect your ability to install OpenShift Dedicated clusters. If you use certain cluster configurations, deploy your cluster in certain AWS regions, or run multiple clusters from your account, you might need to request additional resources for your AWS account. The following table summarizes the AWS components whose limits can impact your ability to install and run OpenShift Dedicated clusters. Component Number of clusters available by default Default AWS limit Description Instance Limits Varies Varies At a minimum, each cluster creates the following instances: One bootstrap machine, which is removed after installation Three control plane nodes Two infrastructure nodes for a single availability zone; three infrascture nodes for multi-availability zones Two worker nodes for a single availability zone; three worker nodes for multi-availability zones These instance type counts are within a new account's default limit. To deploy more worker nodes, deploy large workloads, or use a different instance type, review your account limits to ensure that your cluster can deploy the machines that you need. In most regions, the bootstrap and worker machines uses an m4.large machines and the control plane machines use m4.xlarge instances. In some regions, including all regions that do not support these instance types, m5.large and m5.xlarge instances are used instead. Elastic IPs (EIPs) 0 to 1 5 EIPs per account To provision the cluster in a highly available configuration, the installation program creates a public and private subnet for each availability zone within a region . Each private subnet requires a NAT Gateway , and each NAT gateway requires a separate elastic IP . Review the AWS region map to determine how many availability zones are in each region. To take advantage of the default high availability, install the cluster in a region with at least three availability zones. To install a cluster in a region with more than five availability zones, you must increase the EIP limit. Important To use the us-east-1 region, you must increase the EIP limit for your account. Virtual Private Clouds (VPCs) 5 5 VPCs per region Each cluster creates its own VPC. Elastic Load Balancing (ELB) 3 20 per region By default, each cluster creates internal and external Network Load Balancers for the primary API server and a single Classic Load Balancer for the router. Deploying more Kubernetes LoadBalancer Service objects will create additional load balancers . NAT Gateways 5 5 per availability zone The cluster deploys one NAT gateway in each availability zone. Elastic Network Interfaces (ENIs) At least 12 350 per region The default installation creates 21 ENIs and an ENI for each availability zone in your region. For example, the us-east-1 region contains six availability zones, so a cluster that is deployed in that zone uses 27 ENIs. Review the AWS region map to determine how many availability zones are in each region. Additional ENIs are created for additional machines and load balancers that are created by cluster usage and deployed workloads. VPC Gateway 20 20 per account Each cluster creates a single VPC Gateway for S3 access. S3 buckets 99 100 buckets per account Because the installation process creates a temporary bucket and the registry component in each cluster creates a bucket, you can create only 99 OpenShift Dedicated clusters per AWS account. Security Groups 250 2,500 per account Each cluster creates 10 distinct security groups.

Commands:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "autoscaling:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "elasticloadbalancing:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "cloudwatch:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "events:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "logs:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "support:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "kms:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "sts:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "tag:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "route53:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "servicequotas:ListServices", "servicequotas:GetRequestedServiceQuotaChange", "servicequotas:GetServiceQuota", "servicequotas:RequestServiceQuotaIncrease", "servicequotas:ListServiceQuotas" ], "Resource": [ "*" ] } ] }
{ "Version": "2012-10-17", "Statement": [ { "Action": "*", "Resource": "*", "Effect": "Allow" } ] }
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:AttachVpnGateway", "ec2:DescribeVpnConnections", "ec2:AcceptVpcPeeringConnection", "ec2:DeleteVpcPeeringConnection", "ec2:DescribeVpcPeeringConnections", "ec2:CreateVpnConnectionRoute", "ec2:RejectVpcPeeringConnection", "ec2:DetachVpnGateway", "ec2:DeleteVpnConnectionRoute", "ec2:DeleteVpnGateway", "ec2:DescribeVpcs", "ec2:CreateVpnGateway", "ec2:ModifyVpcPeeringConnectionOptions", "ec2:DeleteVpnConnection", "ec2:CreateVpcPeeringConnection", "ec2:DescribeVpnGateways", "ec2:CreateVpnConnection", "ec2:DescribeRouteTables", "ec2:CreateTags", "ec2:CreateRoute", "directconnect:*" ], "Resource": "*" } ] }
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "aws-portal:ViewAccount", "aws-portal:ViewBilling" ], "Resource": "*" } ] }