Chapter 11. Deploying ROSA without AWS STS 11.1. AWS prerequisites for ROSA Red Hat OpenShift Service on AWS (ROSA) provides a model that allows Red Hat to deploy clusters into a customer's existing Amazon Web Service (AWS) account. You must ensure that the prerequisites are met before installing ROSA. This requirements document does not apply to AWS Security Token Service (STS). If you are using STS, see the STS-specific requirements . Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.1.1. Customer Requirements Red Hat OpenShift Service on AWS (ROSA) clusters must meet several prerequisites before they can be deployed. Note In order to create the cluster, the user must be logged in as an IAM user and not an assumed role or STS user. 11.1.1.1. Account The customer ensures that the AWS limits are sufficient to support Red Hat OpenShift Service on AWS provisioned within the customer's AWS account. The customer's AWS account should be in the customer's AWS Organizations with the applicable service control policy (SCP) applied. Note It is not a requirement that the customer's account be within the AWS Organizations or for the SCP to be applied, however Red Hat must be able to perform all the actions listed in the SCP without restriction. The customer's AWS account should not be transferable to Red Hat. The customer may not impose AWS usage restrictions on Red Hat activities. Imposing restrictions will severely hinder Red Hat's ability to respond to incidents. The customer may deploy native AWS services within the same AWS account. Note Customers are encouraged, but not mandated, to deploy resources in a Virtual Private Cloud (VPC) separate from the VPC hosting Red Hat OpenShift Service on AWS and other Red Hat supported services. 11.1.1.2. Access requirements To appropriately manage the Red Hat OpenShift Service on AWS service, Red Hat must have the AdministratorAccess policy applied to the administrator role at all times. This requirement does not apply if you are using AWS Security Token Service (STS). Note This policy only provides Red Hat with permissions and capabilities to change resources in the customer-provided AWS account. Red Hat must have AWS console access to the customer-provided AWS account. This access is protected and managed by Red Hat. The customer must not utilize the AWS account to elevate their permissions within the Red Hat OpenShift Service on AWS cluster. Actions available in the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa , or OpenShift Cluster Manager console must not be directly performed in the customer's AWS account. 11.1.1.3. Support requirements Red Hat recommends that the customer have at least Business Support from AWS. Red Hat has authority from the customer to request AWS support on their behalf. Red Hat has authority from the customer to request AWS resource limit increases on the customer's account. Red Hat manages the restrictions, limitations, expectations, and defaults for all Red Hat OpenShift Service on AWS clusters in the same manner, unless otherwise specified in this requirements section. 11.1.1.4. Security requirements Volume snapshots will remain within the customer's AWS account and customer-specified region. Red Hat must have ingress access to EC2 hosts and the API server from allow-listed IP addresses. Red Hat must have egress allowed to forward system and audit logs to a Red Hat managed central logging stack. 11.1.2. Required customer procedure Complete these steps before deploying Red Hat OpenShift Service on AWS (ROSA). Procedure If you, as the customer, are utilizing AWS Organizations, then you must use an AWS account within your organization or create a new one . To ensure that Red Hat can perform necessary actions, you must either create a service control policy (SCP) or ensure that none is applied to the AWS account. Attach the SCP to the AWS account. Follow the ROSA procedures for setting up the environment. 11.1.2.1. Minimum set of effective permissions for service control policies (SCP) Service control policies (SCP) are a type of organization policy that manages permissions within your organization. SCPs ensure that accounts within your organization stay within your defined access control guidelines. These policies are maintained in AWS Organizations and control the services that are available within the attached AWS accounts. SCP management is the responsibility of the customer. Note The minimum SCP requirement does not apply when using AWS Security Token Service (STS). For more information about STS, see AWS prerequisites for ROSA with STS . Verify that your service control policy (SCP) does not restrict any of these required permissions. Service Actions Effect Required Amazon EC2 All Allow Amazon EC2 Auto Scaling All Allow Amazon S3 All Allow Identity And Access Management All Allow Elastic Load Balancing All Allow Elastic Load Balancing V2 All Allow Amazon CloudWatch All Allow Amazon CloudWatch Events All Allow Amazon CloudWatch Logs All Allow AWS EC2 Instance Connect SendSerialConsoleSSHPublicKey Allow AWS Support All Allow AWS Key Management Service All Allow AWS Security Token Service All Allow AWS Tiro CreateQuery GetQueryAnswer GetQueryExplanation Allow AWS Marketplace Subscribe Unsubscribe View Subscriptions Allow AWS Resource Tagging All Allow AWS Route53 DNS All Allow AWS Service Quotas ListServices GetRequestedServiceQuotaChange GetServiceQuota RequestServiceQuotaIncrease ListServiceQuotas Allow Optional AWS Billing ViewAccount Viewbilling ViewUsage Allow AWS Cost and Usage Report All Allow AWS Cost Explorer Services All Allow Additional resources Service control policies SCP effects on permissions 11.1.3. Red Hat managed IAM references for AWS Red Hat is responsible for creating and managing the following Amazon Web Services (AWS) resources: IAM policies, IAM users, and IAM roles. 11.1.3.1. IAM Policies Note IAM policies are subject to modification as the capabilities of Red Hat OpenShift Service on AWS change. The AdministratorAccess policy is used by the administration role. This policy provides Red Hat the access necessary to administer the Red Hat OpenShift Service on AWS (ROSA) cluster in the customer's AWS account. 11.1.3.2. IAM users The osdManagedAdmin user is created immediately after installing ROSA into the customer's AWS account. 11.1.4. Provisioned AWS Infrastructure This is an overview of the provisioned Amazon Web Services (AWS) components on a deployed Red Hat OpenShift Service on AWS (ROSA) cluster. 11.1.4.1. EC2 instances AWS EC2 instances are required to deploy the control plane and data plane functions for Red Hat OpenShift Service on AWS. Instance types can vary for control plane and infrastructure nodes, depending on the worker node count. At a minimum, the following EC2 instances are deployed: Three m5.2xlarge control plane nodes Two r5.xlarge infrastructure nodes Two m5.xlarge worker nodes The instance type shown for worker nodes is the default value, but you can customize the instance type for worker nodes according to the needs of your workload. For further guidance on worker node counts, see the information about initial planning considerations in the "Limits and scalability" topic listed in the "Additional resources" section of this page. 11.1.4.2. Amazon Elastic Block Store storage Amazon Elastic Block Store (Amazon EBS) block storage is used for both local node storage and persistent volume storage. The following values are the default size of the local, ephemeral storage provisioned for each EC2 instance. Volume requirements for each EC2 instance: Control Plane Volume Size: 350GB Type: gp3 Input/Output Operations Per Second: 1000 Infrastructure Volume Size: 300GB Type: gp3 Input/Output Operations Per Second: 900 Worker Volume Default size: 300GB Minimum size: 128GB Minimum size: 75GB Type: gp3 Input/Output Operations Per Second: 900 Note Clusters deployed before the release of OpenShift Container Platform 4.11 use gp2 type storage by default. 11.1.4.3. Elastic Load Balancing Each cluster can use up to two Classic Load Balancers for application router and up to two Network Load Balancers for API. For more information, see the ELB documentation for AWS . 11.1.4.4. S3 storage The image registry is backed by AWS S3 storage. Resources Pruning of resources is performed regularly to optimize S3 usage and cluster performance. Note Two buckets are required with a typical size of 2TB each. 11.1.4.5. VPC Configure your VPC according to the following requirements: Subnets : Two subnets for a cluster with a single availability zone, or six subnets for a cluster with multiple availability zones. Red Hat strongly recommends using unique subnets for each cluster. Sharing subnets between multiple clusters is not recommended. Note A public subnet connects directly to the internet through an internet gateway. A private subnet connects to the internet through a network address translation (NAT) gateway. Route tables : One route table per private subnet, and one additional table per cluster. Internet gateways : One Internet Gateway per cluster. NAT gateways : One NAT Gateway per public subnet. Figure 11.1. Sample VPC Architecture 11.1.4.6. Security groups AWS security groups provide security at the protocol and port access level; they are associated with EC2 instances and Elastic Load Balancing (ELB) load balancers. Each security group contains a set of rules that filter traffic coming in and out of one or more EC2 instances. Ensure that the ports required for cluster installation and operation are open on your network and configured to allow access between hosts. The requirements for the default security groups are listed in Required ports for default security groups . Table 11.1. Required ports for default security groups Group Type IP Protocol Port range MasterSecurityGroup AWS::EC2::SecurityGroup icmp 0 tcp 22 tcp 6443 tcp 22623 WorkerSecurityGroup AWS::EC2::SecurityGroup icmp 0 tcp 22 BootstrapSecurityGroup AWS::EC2::SecurityGroup tcp 22 tcp 19531 11.1.4.6.1. Additional custom security groups When you create a cluster using an existing non-managed VPC, you can add additional custom security groups during cluster creation. Custom security groups are subject to the following limitations: You must create the custom security groups in AWS before you create the cluster. For more information, see Amazon EC2 security groups for Linux instances . You must associate the custom security groups with the VPC that the cluster will be installed into. Your custom security groups cannot be associated with another VPC. You might need to request additional quota for your VPC if you are adding additional custom security groups. For information on AWS quota requirements for ROSA, see Required AWS service quotas in Prepare your environment . For information on requesting an AWS quota increase, see Requesting a quota increase . 11.1.5. Networking prerequisites 11.1.5.1. Minimum bandwidth During cluster deployment, Red Hat OpenShift Service on AWS requires a minimum bandwidth of 120 Mbps between cluster resources and public internet resources. When network connectivity is slower than 120 Mbps (for example, when connecting through a proxy) the cluster installation process times out and deployment fails. After deployment, network requirements are determined by your workload. However, a minimum bandwidth of 120 Mbps helps to ensure timely cluster and operator upgrades. 11.1.5.2. AWS firewall prerequisites If you are using a firewall to control egress traffic from Red Hat OpenShift Service on AWS, you must configure your firewall to grant access to the certain domain and port combinations below. Red Hat OpenShift Service on AWS requires this access to provide a fully managed OpenShift service. Important Only ROSA clusters deployed with PrivateLink can use a firewall to control egress traffic. Prerequisites You have configured an Amazon S3 gateway endpoint in your AWS Virtual Private Cloud (VPC). This endpoint is required to complete requests from the cluster to the Amazon S3 service. Procedure Allowlist the following URLs that are used to install and download packages and tools: Domain Port Function registry.redhat.io 443 Provides core container images. quay.io 443 Provides core container images. cdn01.quay.io 443 Provides core container images. cdn02.quay.io 443 Provides core container images. cdn03.quay.io 443 Provides core container images. cdn04.quay.io 443 Provides core container images. cdn05.quay.io 443 Provides core container images. cdn06.quay.io 443 Provides core container images. sso.redhat.com 443 Required. The https://console.redhat.com/openshift site uses authentication from sso.redhat.com to download the pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, chargeback reporting, and so on. quay-registry.s3.amazonaws.com 443 Provides core container images. quayio-production-s3.s3.amazonaws.com 443 Provides core container images. registry.access.redhat.com 443 Hosts all the container images that are stored on the Red Hat Ecosytem Catalog. Additionally, the registry provides access to the odo CLI tool that helps developers build on OpenShift and Kubernetes. access.redhat.com 443 Required. Hosts a signature store that a container client requires for verifying images when pulling them from registry.access.redhat.com . registry.connect.redhat.com 443 Required for all third-party images and certified Operators. console.redhat.com 443 Required. Allows interactions between the cluster and OpenShift Console Manager to enable functionality, such as scheduling upgrades. sso.redhat.com 443 The https://console.redhat.com/openshift site uses authentication from sso.redhat.com . pull.q1w2.quay.rhcloud.com 443 Provides core container images as a fallback when quay.io is not available. catalog.redhat.com 443 The registry.access.redhat.com and https://registry.redhat.io sites redirect through catalog.redhat.com . oidc.op1.openshiftapps.com 443 Used by ROSA for STS implementation with managed OIDC configuration. Allowlist the following telemetry URLs: Domain Port Function cert-api.access.redhat.com 443 Required for telemetry. api.access.redhat.com 443 Required for telemetry. infogw.api.openshift.com 443 Required for telemetry. console.redhat.com 443 Required for telemetry and Red Hat Insights. observatorium-mst.api.openshift.com 443 Required for managed OpenShift-specific telemetry. observatorium.api.openshift.com 443 Required for managed OpenShift-specific telemetry. Managed clusters require enabling telemetry to allow Red Hat to react more quickly to problems, better support the customers, and better understand how product upgrades impact clusters. For more information about how remote health monitoring data is used by Red Hat, see About remote health monitoring in the Additional resources section. Allowlist the following Amazon Web Services (AWS) API URls: Domain Port Function .amazonaws.com 443 Required to access AWS services and resources. Alternatively, if you choose to not use a wildcard for Amazon Web Services (AWS) APIs, you must allowlist the following URLs: Domain Port Function ec2.amazonaws.com 443 Used to install and manage clusters in an AWS environment. events.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment. iam.amazonaws.com 443 Used to install and manage clusters in an AWS environment. route53.amazonaws.com 443 Used to install and manage clusters in an AWS environment. sts.amazonaws.com 443 Used to install and manage clusters in an AWS environment, for clusters configured to use the global endpoint for AWS STS. sts.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment, for clusters configured to use regionalized endpoints for AWS STS. See AWS STS regionalized endpoints for more information. tagging.us-east-1.amazonaws.com 443 Used to install and manage clusters in an AWS environment. This endpoint is always us-east-1, regardless of the region the cluster is deployed in. ec2.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment. elasticloadbalancing.<aws_region>.amazonaws.com 443 Used to install and manage clusters in an AWS environment. tagging.<aws_region>.amazonaws.com 443 Allows the assignment of metadata about AWS resources in the form of tags. Allowlist the following OpenShift URLs: Domain Port Function mirror.openshift.com 443 Used to access mirrored installation content and images. This site is also a source of release image signatures. api.openshift.com 443 Used to check if updates are available for the cluster. Allowlist the following site reliability engineering (SRE) and management URLs: Domain Port Function api.pagerduty.com 443 This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on. events.pagerduty.com 443 This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on. api.deadmanssnitch.com 443 Alerting service used by Red Hat OpenShift Service on AWS to send periodic pings that indicate whether the cluster is available and running. nosnch.in 443 Alerting service used by Red Hat OpenShift Service on AWS to send periodic pings that indicate whether the cluster is available and running. http-inputs-osdsecuritylogs.splunkcloud.com 443 Required. Used by the splunk-forwarder-operator as a logging forwarding endpoint to be used by Red Hat SRE for log-based alerting. sftp.access.redhat.com (Recommended) 22 The SFTP server used by must-gather-operator to upload diagnostic logs to help troubleshoot issues with the cluster. Additional resources About remote health monitoring Security groups Required AWS service quotas 11.1.6. steps Review the required AWS service quotas 11.1.7. Additional resources Limits and scalability SRE access to all Red Hat OpenShift Service on AWS clusters Understanding the ROSA deployment workflow 11.2. Understanding the ROSA deployment workflow Before you create a Red Hat OpenShift Service on AWS (ROSA) cluster, you must complete the AWS prerequisites, verify that the required AWS service quotas are available, and set up your environment. This document provides an overview of the ROSA workflow stages and refers to detailed resources for each stage. Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.2.1. Overview of the ROSA deployment workflow You can follow the workflow stages outlined in this section to set up and access a Red Hat OpenShift Service on AWS (ROSA) cluster. Perform the AWS prerequisites . To deploy a ROSA cluster, your AWS account must meet the prerequisite requirements. Review the required AWS service quotas . To prepare for your cluster deployment, review the AWS service quotas that are required to run a ROSA cluster. Configure your AWS account . Before you create a ROSA cluster, you must enable ROSA in your AWS account, install and configure the AWS CLI ( aws ) tool, and verify the AWS CLI tool configuration. Install the ROSA and OpenShift CLI tools and verify the AWS servce quotas . Install and configure the ROSA CLI ( rosa ) and the OpenShift CLI ( oc ). You can verify if the required AWS resource quotas are available by using the ROSA CLI. Create a ROSA cluster or Create a ROSA cluster using AWS PrivateLink . Use the ROSA CLI ( rosa ) to create a cluster. You can optionally create a ROSA cluster with AWS PrivateLink. Access a cluster . You can configure an identity provider and grant cluster administrator privileges to the identity provider users as required. You can also access a newly deployed cluster quickly by configuring a cluster-admin user. Revoke access to a ROSA cluster for a user . You can revoke access to a ROSA cluster from a user by using the ROSA CLI or the web console. Delete a ROSA cluster . You can delete a ROSA cluster by using the ROSA CLI ( rosa ). 11.2.2. Additional resources For information about using the ROSA deployment workflow to create a cluster that uses the AWS Security Token Service (STS), see Understanding the ROSA with STS deployment workflow . Configuring identity providers Deleting a cluster Deleting access to a cluster Command quick reference for creating clusters and users 11.3. Required AWS service quotas Review this list of the required Amazon Web Service (AWS) service quotas that are required to run an Red Hat OpenShift Service on AWS cluster. Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.3.1. Required AWS service quotas The table below describes the AWS service quotas and levels required to create and run one Red Hat OpenShift Service on AWS cluster. Although most default values are suitable for most workloads, you might need to request additional quota for the following cases: ROSA clusters require a minimum AWS EC2 service quota of 100 vCPUs to provide for cluster creation, availability, and upgrades. The default maximum value for vCPUs assigned to Running On-Demand Standard Amazon EC2 instances is 5 . Therefore if you have not created a ROSA cluster using the same AWS account previously, you must request additional EC2 quota for Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances . Some optional cluster configuration features, such as custom security groups, might require you to request additional quota. For example, because ROSA associates 1 security group with network interfaces in worker machine pools by default, and the default quota for Security groups per network interface is 5 , if you want to add 5 custom security groups, you must request additional quota, because this would bring the total number of security groups on worker network interfaces to 6. Note The AWS SDK allows ROSA to check quotas, but the AWS SDK calculation does not account for your existing usage. Therefore, it is possible that the quota check can pass in the AWS SDK yet the cluster creation can fail. To fix this issue, increase your quota. If you need to modify or increase a specific quota, see Amazon's documentation on requesting a quota increase . Large quota requests are submitted to Amazon Support for review, and take some time to be approved. If your quota request is urgent, contact AWS Support. Table 11.2. ROSA-required service quota Quota name Service code Quota code AWS default Minimum required Description Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances ec2 L-1216C47A 5 100 Maximum number of vCPUs assigned to the Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances. The default value of 5 vCPUs is not sufficient to create ROSA clusters. Storage for General Purpose SSD (gp2) volume storage in TiB ebs L-D18FCD1D 50 300 The maximum aggregated amount of storage, in TiB, that can be provisioned across General Purpose SSD (gp2) volumes in this Region. Storage for General Purpose SSD (gp3) volume storage in TiB ebs L-7A658B76 50 300 The maximum aggregated amount of storage, in TiB, that can be provisioned across General Purpose SSD (gp3) volumes in this Region. 300 TiB of storage is the required minimum for optimal performance. Storage for Provisioned IOPS SSD (io1) volumes in TiB ebs L-FD252861 50 300 The maximum aggregated amount of storage, in TiB, that can be provisioned across Provisioned IOPS SSD (io1) volumes in this Region. 300 TiB of storage is the required minimum for optimal performance. Table 11.3. General AWS service quotas Quota name Service code Quota code AWS default Minimum required Description EC2-VPC Elastic IPs ec2 L-0263D0A3 5 5 The maximum number of Elastic IP addresses that you can allocate for EC2-VPC in this Region. VPCs per Region vpc L-F678F1CE 5 5 The maximum number of VPCs per Region. This quota is directly tied to the maximum number of internet gateways per Region. Internet gateways per Region vpc L-A4707A72 5 5 The maximum number of internet gateways per Region. This quota is directly tied to the maximum number of VPCs per Region. To increase this quota, increase the number of VPCs per Region. Network interfaces per Region vpc L-DF5E4CA3 5,000 5,000 The maximum number of network interfaces per Region. Security groups per network interface vpc L-2AFB9258 5 5 The maximum number of security groups per network interface. This quota, multiplied by the quota for rules per security group, cannot exceed 1000. Snapshots per Region ebs L-309BACF6 10,000 10,000 The maximum number of snapshots per Region IOPS for Provisioned IOPS SSD (Io1) volumes ebs L-B3A130E6 300,000 300,000 The maximum aggregated number of IOPS that can be provisioned across Provisioned IOPS SDD (io1) volumes in this Region. Application Load Balancers per Region elasticloadbalancing L-53DA6B97 50 50 The maximum number of Application Load Balancers that can exist in each region. Classic Load Balancers per Region elasticloadbalancing L-E9E9831D 20 20 The maximum number of Classic Load Balancers that can exist in each region. 11.3.1.1. Additional resources How can I request, view, and manage service quota increase requests using AWS CLI commands? ROSA service quotas Request a quota increase IAM and AWS STS quotas (AWS documentation) 11.3.2. steps Configure your AWS account 11.3.3. Additional resources Understanding the ROSA deployment workflow 11.4. Configuring your AWS account After you complete the AWS prerequisites, configure your AWS account and enable the Red Hat OpenShift Service on AWS (ROSA) service. Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.4.1. Configuring your AWS account To configure your AWS account to use the ROSA service, complete the following steps. Prerequisites Review and complete the deployment prerequisites and policies. Create a Red Hat account , if you do not already have one. Then, check your email for a verification link. You will need these credentials to install ROSA. Procedure Log in to the Amazon Web Services (AWS) account that you want to use. A dedicated AWS account is recommended to run production clusters. If you are using AWS Organizations, you can use an AWS account within your organization or create a new one . If you are using AWS Organizations and you need to have a service control policy (SCP) applied to the AWS account you plan to use, see AWS Prerequisites for details on the minimum required SCP. As part of the cluster creation process, rosa establishes an osdCcsAdmin IAM user. This user uses the IAM credentials you provide when configuring the AWS CLI. Note This user has Programmatic access enabled and the AdministratorAccess policy attached to it. Enable the ROSA service in the AWS Console. Sign in to your AWS account . To enable ROSA, go to the ROSA service and select Enable OpenShift . Install and configure the AWS CLI. Follow the AWS command-line interface documentation to install and configure the AWS CLI for your operating system. Specify the correct aws_access_key_id and aws_secret_access_key in the .aws/credentials file. See AWS Configuration basics in the AWS documentation. Set a default AWS region. Note It is recommended to set the default AWS region by using the environment variable. The ROSA service evaluates regions in the following priority order: The region specified when running the rosa command with the --region flag. The region set in the AWS_DEFAULT_REGION environment variable. See Environment variables to configure the AWS CLI in the AWS documentation. The default region set in your AWS configuration file. See Quick configuration with aws configure in the AWS documentation. Optional: Configure your AWS CLI settings and credentials by using an AWS named profile. rosa evaluates AWS named profiles in the following priority order: The profile specified when running the rosa command with the --profile flag. The profile set in the AWS_PROFILE environment variable. See Named profiles in the AWS documentation. Verify the AWS CLI is installed and configured correctly by running the following command to query the AWS API: USD aws sts get-caller-identity --output text Example output <aws_account_id> arn:aws:iam::<aws_account_id>:user/<username> <aws_user_id> After completing these steps, install ROSA. 11.4.2. steps Installing the ROSA CLI 11.4.3. Additional resources AWS prerequisites Required AWS service quotas and requesting increases Understanding the ROSA deployment workflow 11.5. Installing the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa After you configure your AWS account, install and configure the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa . Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.5.1. Installing and configuring the ROSA CLI Install and configure the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa . You can also install the OpenShift CLI ( oc ) and verify if the required AWS resource quotas are available by using the ROSA CLI ( rosa ). Prerequisites Review and complete the AWS prerequisites and ROSA policies. Create a Red Hat account , if you do not already have one. Then, check your email for a verification link. You will need these credentials to install ROSA. Configure your AWS account and enable the ROSA service in your AWS account. Procedure Install rosa , the Red Hat OpenShift Service on AWS command-line interface (CLI). Download the latest release of the ROSA CLI for your operating system. Optional: Rename the executable file you downloaded to rosa . This documentation uses rosa to refer to the executable file. Optional: Add rosa to your path. Example USD mv rosa /usr/local/bin/rosa Enter the following command to verify your installation: USD rosa Example output Command line tool for Red Hat OpenShift Service on AWS. For further documentation visit https://access.redhat.com/documentation/en-us/red_hat_openshift_service_on_aws Usage: rosa [command] Available Commands: completion Generates completion scripts create Create a resource from stdin delete Delete a specific resource describe Show details of a specific resource download Download necessary tools for using your cluster edit Edit a specific resource grant Grant role to a specific resource help Help about any command init Applies templates to support Red Hat OpenShift Service on AWS install Installs a resource into a cluster link Link a ocm/user role from stdin list List all resources of a specific type login Log in to your Red Hat account logout Log out logs Show installation or uninstallation logs for a cluster revoke Revoke role from a specific resource uninstall Uninstalls a resource from a cluster unlink UnLink a ocm/user role from stdin upgrade Upgrade a resource verify Verify resources are configured correctly for cluster install version Prints the version of the tool whoami Displays user account information Flags: --color string Surround certain characters with escape sequences to display them in color on the terminal. Allowed options are [auto never always] (default "auto") --debug Enable debug mode. -h, --help help for rosa Use "rosa [command] --help" for more information about a command. Optional: Generate the command completion scripts for the ROSA CLI. The following example generates the Bash completion scripts for a Linux machine: USD rosa completion bash | sudo tee /etc/bash_completion.d/rosa Optional: Enable command completion for the ROSA CLI from your existing terminal. The following example enables Bash completion for rosa in an existing terminal on a Linux machine: USD source /etc/bash_completion.d/rosa Log in to your Red Hat account with rosa . Enter the following command. USD rosa login Replace <my_offline_access_token> with your token. Example output To login to your Red Hat account, get an offline access token at https://console.redhat.com/openshift/token/rosa ? Copy the token and paste it here: <my-offline-access-token> Example output continued I: Logged in as 'rh-rosa-user' on 'https://api.openshift.com' Enter the following command to verify that your AWS account has the necessary permissions. USD rosa verify permissions Example output I: Validating SCP policies... I: AWS SCP policies ok Note This command verifies permissions only for ROSA clusters that do not use the AWS Security Token Service (STS). Verify that your AWS account has the necessary quota to deploy an Red Hat OpenShift Service on AWS cluster. USD rosa verify quota --region=us-west-2 Example output I: Validating AWS quota... I: AWS quota ok Note Sometimes your AWS quota varies by region. If you receive any errors, try a different region. If you need to increase your quota, go to your AWS console , and request a quota increase for the service that failed. After both the permissions and quota checks pass, proceed to the step. Prepare your AWS account for cluster deployment: Run the following command to verify your Red Hat and AWS credentials are setup correctly. Check that your AWS Account ID, Default Region and ARN match what you expect. You can safely ignore the rows beginning with OCM for now. USD rosa whoami Example output AWS Account ID: 000000000000 AWS Default Region: us-east-2 AWS ARN: arn:aws:iam::000000000000:user/hello OCM API: https://api.openshift.com OCM Account ID: 1DzGIdIhqEWyt8UUXQhSoWaaaaa OCM Account Name: Your Name OCM Account Username: you@domain.com OCM Account Email: you@domain.com OCM Organization ID: 1HopHfA2hcmhup5gCr2uH5aaaaa OCM Organization Name: Red Hat OCM Organization External ID: 0000000 Initialize your AWS account. This step runs a CloudFormation template that prepares your AWS account for cluster deployment and management. This step typically takes 1-2 minutes to complete. USD rosa init Example output I: Logged in as 'rh-rosa-user' on 'https://api.openshift.com' I: Validating AWS credentials... I: AWS credentials are valid! I: Validating SCP policies... I: AWS SCP policies ok I: Validating AWS quota... I: AWS quota ok I: Ensuring cluster administrator user 'osdCcsAdmin'... I: Admin user 'osdCcsAdmin' created successfully! I: Verifying whether OpenShift command-line tool is available... E: OpenShift command-line tool is not installed. Run 'rosa download oc' to download the latest version, then add it to your PATH. Install the OpenShift CLI ( oc ) from the ROSA CLI. Enter this command to download the latest version of the oc CLI: USD rosa download oc After downloading the oc CLI, unzip it and add it to your path. Enter this command to verify that the oc CLI is installed correctly: USD rosa verify oc After installing ROSA, you are ready to create a cluster. 11.5.2. steps Create a ROSA cluster or Create an AWS PrivateLink cluster on ROSA . 11.5.3. Additional resources AWS prerequisites Required AWS service quotas and requesting increases Understanding the ROSA deployment workflow 11.6. Creating a ROSA cluster without AWS STS After you set up your environment and install Red Hat OpenShift Service on AWS (ROSA), create a cluster. This document describes how to set up a ROSA cluster. Alternatively, you can create a ROSA cluster with AWS PrivateLink. Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.6.1. Creating your cluster You can create a Red Hat OpenShift Service on AWS (ROSA) cluster using the ROSA CLI ( rosa ). Prerequisites You have installed Red Hat OpenShift Service on AWS. Note AWS Shared VPCs are not currently supported for ROSA installs. Procedure You can create a cluster using the default settings or by specifying custom settings using the interactive mode. To view other options when creating a cluster, enter the rosa create cluster --help command. Creating a cluster can take up to 40 minutes. Note Multiple availability zones (AZ) are recommended for production workloads. The default is a single availability zone. Use --help for an example of how to set this option manually or use interactive mode to be prompted for this setting. To create your cluster with the default cluster settings: USD rosa create cluster --cluster-name=<cluster_name> Example output I: Creating cluster with identifier '1de87g7c30g75qechgh7l5b2bha6r04e' and name 'rh-rosa-test-cluster1' I: To view list of clusters and their status, run `rosa list clusters` I: Cluster 'rh-rosa-test-cluster1' has been created. I: Once the cluster is 'Ready' you will need to add an Identity Provider and define the list of cluster administrators. See `rosa create idp --help` and `rosa create user --help` for more information. I: To determine when your cluster is Ready, run `rosa describe cluster rh-rosa-test-cluster1`. To create a cluster using interactive prompts: USD rosa create cluster --interactive To configure your networking IP ranges, you can use the following default ranges. For more information when using manual mode, use the rosa create cluster --help | grep cidr command. In interactive mode, you are prompted for the settings. Node CIDR: 10.0.0.0/16 Service CIDR: 172.30.0.0/16 Pod CIDR: 10.128.0.0/14 Enter the following command to check the status of your cluster. During cluster creation, the State field from the output will transition from pending to installing , and finally to ready . USD rosa describe cluster --cluster=<cluster_name> Example output Name: rh-rosa-test-cluster1 OpenShift Version: 4.6.8 DNS: *.example.com ID: uniqueidnumber External ID: uniqueexternalidnumber AWS Account: 123456789101 API URL: https://api.rh-rosa-test-cluster1.example.org:6443 Console URL: https://console-openshift-console.apps.rh-rosa-test-cluster1.example.or Nodes: Master: 3, Infra: 2, Compute: 2 Region: us-west-2 Multi-AZ: false State: ready Channel Group: stable Private: No Created: Jan 15 2021 16:30:55 UTC Details Page: https://console.redhat.com/examplename/details/idnumber Note If installation fails or the State field does not change to ready after 40 minutes, check the installation troubleshooting documentation for more details. Track the progress of the cluster creation by watching the OpenShift installer logs: USD rosa logs install --cluster=<cluster_name> --watch 11.6.2. steps Configure identity providers 11.6.3. Additional resources Understanding the ROSA deployment workflow Deleting a ROSA cluster ROSA architecture models 11.7. Configuring a private cluster A Red Hat OpenShift Service on AWS cluster can be made private so that internal applications can be hosted inside a corporate network. In addition, private clusters can be configured to have only internal API endpoints for increased security. Privacy settings can be configured during cluster creation or after a cluster is established. 11.7.1. Enabling private cluster on a new cluster You can enable the private cluster setting when creating a new Red Hat OpenShift Service on AWS cluster. Important Private clusters cannot be used with AWS security token service (STS). However, STS supports AWS PrivateLink clusters. Prerequisites AWS VPC Peering, VPN, DirectConnect, or TransitGateway has been configured to allow private access. Procedure Enter the following command to create a new private cluster. USD rosa create cluster --cluster-name=<cluster_name> --private Note Alternatively, use --interactive to be prompted for each cluster option. 11.7.2. Enabling private cluster on an existing cluster After a cluster has been created, you can later enable the cluster to be private. Important Private clusters cannot be used with AWS security token service (STS). However, STS supports AWS PrivateLink clusters. Prerequisites AWS VPC Peering, VPN, DirectConnect, or TransitGateway has been configured to allow private access. Procedure Enter the following command to enable the --private option on an existing cluster. USD rosa edit cluster --cluster=<cluster_name> --private Note Transitioning your cluster between private and public can take several minutes to complete. 11.7.3. Additional resources Creating an AWS PrivateLink cluster on ROSA 11.8. Deleting access to a ROSA cluster Delete access to a Red Hat OpenShift Service on AWS (ROSA) cluster using the rosa command-line. Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.8.1. Revoking dedicated-admin access using the ROSA CLI You can revoke access for a dedicated-admin user if you are the user who created the cluster, the organization administrator user, or the super administrator user. Prerequisites You have added an Identity Provider (IDP) to your cluster. You have the IDP user name for the user whose privileges you are revoking. You are logged in to the cluster. Procedure Enter the following command to revoke the dedicated-admin access of a user: USD rosa revoke user dedicated-admin --user=<idp_user_name> --cluster=<cluster_name> Enter the following command to verify that your user no longer has dedicated-admin access. The output does not list the revoked user. USD oc get groups dedicated-admins 11.8.2. Revoking cluster-admin access using the ROSA CLI Only the user who created the cluster can revoke access for cluster-admin users. Prerequisites You have added an Identity Provider (IDP) to your cluster. You have the IDP user name for the user whose privileges you are revoking. You are logged in to the cluster. Procedure Enter the following command to revoke the cluster-admin access of a user: USD rosa revoke user cluster-admins --user=myusername --cluster=mycluster Enter the following command to verify that the user no longer has cluster-admin access. The output does not list the revoked user. USD oc get groups cluster-admins 11.9. Deleting a ROSA cluster Delete a Red Hat OpenShift Service on AWS (ROSA) cluster using the rosa command-line. Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.9.1. Prerequisites If Red Hat OpenShift Service on AWS created a VPC, you must remove the following items from your cluster before you can successfully delete your cluster: Network configurations, such as VPN configurations and VPC peering connections Any additional services that were added to the VPC If these configurations and services remain, the cluster does not delete properly. 11.9.2. Deleting a ROSA cluster and the cluster-specific IAM resources You can delete a Red Hat OpenShift Service on AWS (ROSA) with AWS Security Token Service (STS) cluster by using the ROSA CLI ( rosa ) or Red Hat OpenShift Cluster Manager. After deleting the cluster, you can clean up the cluster-specific Identity and Access Management (IAM) resources in your AWS account by using the ROSA CLI ( rosa ). The cluster-specific resources include the Operator roles and the OpenID Connect (OIDC) provider. Note The cluster deletion must complete before you remove the IAM resources, because the resources are used in the cluster deletion and clean-up processes. If add-ons are installed, the cluster deletion takes longer because add-ons are uninstalled before the cluster is deleted. The amount of time depends on the number and size of the add-ons. Important If the cluster that created the VPC during the installation is deleted, the associated installation program-created VPC will also be deleted, resulting in the failure of all the clusters that are using the same VPC. Additionally, any resources created with the same tagSet key-value pair of the resources created by the installation program and labeled with a value of owned will also be deleted. Prerequisites You have installed a ROSA cluster. You have installed and configured the latest ROSA CLI ( rosa ) on your installation host. Procedure Obtain the cluster ID, the Amazon Resource Names (ARNs) for the cluster-specific Operator roles and the endpoint URL for the OIDC provider: USD rosa describe cluster --cluster=<cluster_name> 1 1 Replace <cluster_name> with the name of your cluster. Example output Name: mycluster ID: 1s3v4x39lhs8sm49m90mi0822o34544a 1 ... Operator IAM Roles: 2 - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-machine-api-aws-cloud-credentials - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-cloud-credential-operator-cloud-crede - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-image-registry-installer-cloud-creden - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-cluster-csi-drivers-ebs-cloud-credent - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-cloud-network-config-controller-cloud State: ready Private: No Created: May 13 2022 11:26:15 UTC Details Page: https://console.redhat.com/openshift/details/s/296kyEFwzoy1CREQicFRdZybrc0 OIDC Endpoint URL: https://oidc.op1.openshiftapps.com/<oidc_config_id> 3 1 Lists the cluster ID. 2 Specifies the ARNs for the cluster-specific Operator roles. For example, in the sample output the ARN for the role required by the Machine Config Operator is arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-machine-api-aws-cloud-credentials . 3 Displays the endpoint URL for the cluster-specific OIDC provider. Important You require the cluster ID to delete the cluster-specific STS resources using the ROSA CLI ( rosa ) after the cluster is deleted. Delete the cluster: To delete the cluster by using Red Hat OpenShift Cluster Manager: Navigate to OpenShift Cluster Manager . Click the Options menu to your cluster and select Delete cluster . Type the name of your cluster at the prompt and click Delete . To delete the cluster using the ROSA CLI ( rosa ): Enter the following command to delete the cluster and watch the logs, replacing <cluster_name> with the name or ID of your cluster: USD rosa delete cluster --cluster=<cluster_name> --watch Important You must wait for the cluster deletion to complete before you remove the Operator roles and the OIDC provider. The cluster-specific Operator roles are required to clean-up the resources created by the OpenShift Operators. The Operators use the OIDC provider to authenticate. Delete the OIDC provider that the cluster Operators use to authenticate: USD rosa delete oidc-provider -c <cluster_id> --mode auto 1 1 Replace <cluster_id> with the ID of the cluster. Note You can use the -y option to automatically answer yes to the prompts. Optional. Delete the cluster-specific Operator IAM roles: Important The account-wide IAM roles can be used by other ROSA clusters in the same AWS account. Only remove the roles if they are not required by other clusters. USD rosa delete operator-roles -c <cluster_id> --mode auto 1 1 Replace <cluster_id> with the ID of the cluster. Troubleshooting If the cluster cannot be deleted because of missing IAM roles, see Additional Repairing a cluster that cannot be deleted . If the cluster cannot be deleted for other reasons: Check that there are no Add-ons for your cluster pending in the Hybrid Cloud Console . Check that all AWS resources and dependencies have been deleted in the Amazon Web Console. 11.10. Command quick reference for creating clusters and users Tip AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS because it provides enhanced security. 11.10.1. Command quick reference list If you have already created your first cluster and users, this list can serve as a command quick reference list when creating additional clusters and users. ## Configures your AWS account and ensures everything is setup correctly USD rosa init ## Starts the cluster creation process (~30-40minutes) USD rosa create cluster --cluster-name=<cluster_name> ## Connect your IDP to your cluster USD rosa create idp --cluster=<cluster_name> --interactive ## Promotes a user from your IDP to dedicated-admin level USD rosa grant user dedicated-admin --user=<idp_user_name> --cluster=<cluster_name> ## Checks if your install is ready (look for State: Ready), ## and provides your Console URL to login to the web console. USD rosa describe cluster --cluster=<cluster_name> 11.10.2. Additional resources Understanding the ROSA deployment workflow

Commands:
{ "Version": "2012-10-17", "Statement": [ { "Action": "*", "Resource": "*", "Effect": "Allow" } ] }
aws sts get-caller-identity --output text
<aws_account_id> arn:aws:iam::<aws_account_id>:user/<username> <aws_user_id>
mv rosa /usr/local/bin/rosa
rosa
Command line tool for Red Hat OpenShift Service on AWS. For further documentation visit https://access.redhat.com/documentation/en-us/red_hat_openshift_service_on_aws Usage: rosa [command] Available Commands: completion Generates completion scripts create Create a resource from stdin delete Delete a specific resource describe Show details of a specific resource download Download necessary tools for using your cluster edit Edit a specific resource grant Grant role to a specific resource help Help about any command init Applies templates to support Red Hat OpenShift Service on AWS install Installs a resource into a cluster link Link a ocm/user role from stdin list List all resources of a specific type login Log in to your Red Hat account logout Log out logs Show installation or uninstallation logs for a cluster revoke Revoke role from a specific resource uninstall Uninstalls a resource from a cluster unlink UnLink a ocm/user role from stdin upgrade Upgrade a resource verify Verify resources are configured correctly for cluster install version Prints the version of the tool whoami Displays user account information Flags: --color string Surround certain characters with escape sequences to display them in color on the terminal. Allowed options are [auto never always] (default "auto") --debug Enable debug mode. -h, --help help for rosa Use "rosa [command] --help" for more information about a command.
rosa completion bash | sudo tee /etc/bash_completion.d/rosa
source /etc/bash_completion.d/rosa
rosa login
To login to your Red Hat account, get an offline access token at https://console.redhat.com/openshift/token/rosa ? Copy the token and paste it here: <my-offline-access-token>
I: Logged in as 'rh-rosa-user' on 'https://api.openshift.com'
rosa verify permissions
I: Validating SCP policies I: AWS SCP policies ok
rosa verify quota --region=us-west-2
I: Validating AWS quota I: AWS quota ok
rosa whoami
AWS Account ID: 000000000000 AWS Default Region: us-east-2 AWS ARN: arn:aws:iam::000000000000:user/hello OCM API: https://api.openshift.com OCM Account ID: 1DzGIdIhqEWyt8UUXQhSoWaaaaa OCM Account Name: Your Name OCM Account Username: you@domain.com OCM Account Email: you@domain.com OCM Organization ID: 1HopHfA2hcmhup5gCr2uH5aaaaa OCM Organization Name: Red Hat OCM Organization External ID: 0000000
rosa init
I: Logged in as 'rh-rosa-user' on 'https://api.openshift.com' I: Validating AWS credentials I: AWS credentials are valid! I: Validating SCP policies I: AWS SCP policies ok I: Validating AWS quota I: AWS quota ok I: Ensuring cluster administrator user 'osdCcsAdmin' I: Admin user 'osdCcsAdmin' created successfully! I: Verifying whether OpenShift command-line tool is available E: OpenShift command-line tool is not installed. Run 'rosa download oc' to download the latest version, then add it to your PATH.
rosa download oc
rosa verify oc
rosa create cluster --cluster-name=<cluster_name>
I: Creating cluster with identifier '1de87g7c30g75qechgh7l5b2bha6r04e' and name 'rh-rosa-test-cluster1' I: To view list of clusters and their status, run `rosa list clusters` I: Cluster 'rh-rosa-test-cluster1' has been created. I: Once the cluster is 'Ready' you will need to add an Identity Provider and define the list of cluster administrators. See `rosa create idp --help` and `rosa create user --help` for more information. I: To determine when your cluster is Ready, run `rosa describe cluster rh-rosa-test-cluster1`.
rosa create cluster --interactive
rosa describe cluster --cluster=<cluster_name>
Name: rh-rosa-test-cluster1 OpenShift Version: 4.6.8 DNS: *.example.com ID: uniqueidnumber External ID: uniqueexternalidnumber AWS Account: 123456789101 API URL: https://api.rh-rosa-test-cluster1.example.org:6443 Console URL: https://console-openshift-console.apps.rh-rosa-test-cluster1.example.or Nodes: Master: 3, Infra: 2, Compute: 2 Region: us-west-2 Multi-AZ: false State: ready Channel Group: stable Private: No Created: Jan 15 2021 16:30:55 UTC Details Page: https://console.redhat.com/examplename/details/idnumber
rosa logs install --cluster=<cluster_name> --watch
rosa create cluster --cluster-name=<cluster_name> --private
rosa edit cluster --cluster=<cluster_name> --private
rosa revoke user dedicated-admin --user=<idp_user_name> --cluster=<cluster_name>
oc get groups dedicated-admins
rosa revoke user cluster-admins --user=myusername --cluster=mycluster
oc get groups cluster-admins
rosa describe cluster --cluster=<cluster_name> 1
Name: mycluster ID: 1s3v4x39lhs8sm49m90mi0822o34544a 1 Operator IAM Roles: 2 - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-machine-api-aws-cloud-credentials - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-cloud-credential-operator-cloud-crede - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-image-registry-installer-cloud-creden - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-cluster-csi-drivers-ebs-cloud-credent - arn:aws:iam::<aws_account_id>:role/mycluster-x4q9-openshift-cloud-network-config-controller-cloud State: ready Private: No Created: May 13 2022 11:26:15 UTC Details Page: https://console.redhat.com/openshift/details/s/296kyEFwzoy1CREQicFRdZybrc0 OIDC Endpoint URL: https://oidc.op1.openshiftapps.com/<oidc_config_id> 3
rosa delete cluster --cluster=<cluster_name> --watch
rosa delete oidc-provider -c <cluster_id> --mode auto 1
rosa delete operator-roles -c <cluster_id> --mode auto 1
## Configures your AWS account and ensures everything is setup correctly rosa init ## Starts the cluster creation process (~30-40minutes) rosa create cluster --cluster-name=<cluster_name> ## Connect your IDP to your cluster rosa create idp --cluster=<cluster_name> --interactive ## Promotes a user from your IDP to dedicated-admin level rosa grant user dedicated-admin --user=<idp_user_name> --cluster=<cluster_name> ## Checks if your install is ready (look for State: Ready), ## and provides your Console URL to login to the web console. rosa describe cluster --cluster=<cluster_name>