Chapter 4. Deploying hosted control planes 4.1. Deploying hosted control planes on AWS A hosted cluster is an OpenShift Container Platform cluster with its API endpoint and control plane that are hosted on the management cluster. The hosted cluster includes the control plane and its corresponding data plane. To configure hosted control planes on premises, you must install multicluster engine for Kubernetes Operator in a management cluster. By deploying the HyperShift Operator on an existing managed cluster by using the hypershift-addon managed cluster add-on, you can enable that cluster as a management cluster and start to create the hosted cluster. The hypershift-addon managed cluster add-on is enabled by default for the local-cluster managed cluster. You can use the multicluster engine Operator console or the hosted control plane command-line interface (CLI), hcp , to create a hosted cluster. The hosted cluster is automatically imported as a managed cluster. However, you can disable this automatic import feature into multicluster engine Operator . 4.1.1. Preparing to deploy hosted control planes on AWS As you prepare to deploy hosted control planes on Amazon Web Services (AWS), consider the following information: Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it. Do not use clusters as a hosted cluster name. Run the management cluster and workers on the same platform for hosted control planes. A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster. 4.1.1.1. Prerequisites to configure a management cluster You must have the following prerequisites to configure the management cluster: You have installed the multicluster engine for Kubernetes Operator 2.5 and later on an OpenShift Container Platform cluster. The multicluster engine Operator is automatically installed when you install Red Hat Advanced Cluster Management (RHACM). The multicluster engine Operator can also be installed without RHACM as an Operator from the OpenShift Container Platform OperatorHub. You have at least one managed OpenShift Container Platform cluster for the multicluster engine Operator. The local-cluster is automatically imported in the multicluster engine Operator version 2.5 and later. You can check the status of your hub cluster by running the following command: USD oc get managedclusters local-cluster You have installed the aws command-line interface (CLI) . You have installed the hosted control plane CLI, hcp . Additional resources Configuring Ansible Automation Platform jobs to run on hosted clusters Advanced configuration Enabling the central infrastructure management service Manually enabling the hosted control planes feature Disabling the hosted control planes feature Deploying the SR-IOV Operator for hosted control planes 4.1.2. Creating the Amazon Web Services S3 bucket and S3 OIDC secret Before you can create and manage hosted clusters on Amazon Web Services (AWS), you must create the S3 bucket and S3 OIDC secret. Procedure Create an S3 bucket that has public access to host OIDC discovery documents for your clusters by running the following commands: USD aws s3api create-bucket --bucket <bucket_name> \ 1 --create-bucket-configuration LocationConstraint=<region> \ 2 --region <region> 3 1 Replace <bucket_name> with the name of the S3 bucket you are creating. 2 3 To create the bucket in a region other than the us-east-1 region, include this line and replace <region> with the region you want to use. To create a bucket in the us-east-1 region, omit this line. USD aws s3api delete-public-access-block --bucket <bucket_name> 1 1 Replace <bucket_name> with the name of the S3 bucket you are creating. USD echo '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<bucket_name>/*" 1 } ] }' | envsubst > policy.json 1 Replace <bucket_name> with the name of the S3 bucket you are creating. USD aws s3api put-bucket-policy --bucket <bucket_name> \ 1 --policy file://policy.json 1 Replace <bucket_name> with the name of the S3 bucket you are creating. Note If you are using a Mac computer, you must export the bucket name in order for the policy to work. Create an OIDC S3 secret named hypershift-operator-oidc-provider-s3-credentials for the HyperShift Operator. Save the secret in the local-cluster namespace. See the following table to verify that the secret contains the following fields: Table 4.1. Required fields for the AWS secret Field name Description bucket Contains an S3 bucket with public access to host OIDC discovery documents for your hosted clusters. credentials A reference to a file that contains the credentials of the default profile that can access the bucket. By default, HyperShift only uses the default profile to operate the bucket . region Specifies the region of the S3 bucket. To create an AWS secret, run the following command: USD oc create secret generic <secret_name> \ --from-file=credentials=<path>/.aws/credentials \ --from-literal=bucket=<s3_bucket> \ --from-literal=region=<region> \ -n local-cluster Note Disaster recovery backup for the secret is not automatically enabled. To add the label that enables the hypershift-operator-oidc-provider-s3-credentials secret to be backed up for disaster recovery, run the following command: USD oc label secret hypershift-operator-oidc-provider-s3-credentials \ -n local-cluster cluster.open-cluster-management.io/backup=true 4.1.3. Creating a routable public zone for hosted clusters To access applications in your hosted clusters, you must configure the routable public zone. If the public zone exists, skip this step. Otherwise, the public zone affects the existing functions. Procedure To create a routable public zone for DNS records, enter the following command: USD aws route53 create-hosted-zone \ --name <basedomain> \ 1 --caller-reference USD(whoami)-USD(date --rfc-3339=date) 1 Replace <basedomain> with your base domain, for example, www.example.com . 4.1.4. Creating an AWS IAM role and STS credentials Before creating a hosted cluster on Amazon Web Services (AWS), you must create an AWS IAM role and STS credentials. Procedure Get the Amazon Resource Name (ARN) of your user by running the following command: USD aws sts get-caller-identity --query "Arn" --output text Example output arn:aws:iam::1234567890:user/<aws_username> Use this output as the value for <arn> in the step. Create a JSON file that contains the trust relationship configuration for your role. See the following example: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "<arn>" 1 }, "Action": "sts:AssumeRole" } ] } 1 Replace <arn> with the ARN of your user that you noted in the step. Create the Identity and Access Management (IAM) role by running the following command: USD aws iam create-role \ --role-name <name> \ 1 --assume-role-policy-document file://<file_name>.json \ 2 --query "Role.Arn" 1 Replace <name> with the role name, for example, hcp-cli-role . 2 Replace <file_name> with the name of the JSON file you created in the step. Example output arn:aws:iam::820196288204:role/myrole Create a JSON file named policy.json that contains the following permission policies for your role: { "Version": "2012-10-17", "Statement": [ { "Sid": "EC2", "Effect": "Allow", "Action": [ "ec2:CreateDhcpOptions", "ec2:DeleteSubnet", "ec2:ReplaceRouteTableAssociation", "ec2:DescribeAddresses", "ec2:DescribeInstances", "ec2:DeleteVpcEndpoints", "ec2:CreateNatGateway", "ec2:CreateVpc", "ec2:DescribeDhcpOptions", "ec2:AttachInternetGateway", "ec2:DeleteVpcEndpointServiceConfigurations", "ec2:DeleteRouteTable", "ec2:AssociateRouteTable", "ec2:DescribeInternetGateways", "ec2:DescribeAvailabilityZones", "ec2:CreateRoute", "ec2:CreateInternetGateway", "ec2:RevokeSecurityGroupEgress", "ec2:ModifyVpcAttribute", "ec2:DeleteInternetGateway", "ec2:DescribeVpcEndpointConnections", "ec2:RejectVpcEndpointConnections", "ec2:DescribeRouteTables", "ec2:ReleaseAddress", "ec2:AssociateDhcpOptions", "ec2:TerminateInstances", "ec2:CreateTags", "ec2:DeleteRoute", "ec2:CreateRouteTable", "ec2:DetachInternetGateway", "ec2:DescribeVpcEndpointServiceConfigurations", "ec2:DescribeNatGateways", "ec2:DisassociateRouteTable", "ec2:AllocateAddress", "ec2:DescribeSecurityGroups", "ec2:RevokeSecurityGroupIngress", "ec2:CreateVpcEndpoint", "ec2:DescribeVpcs", "ec2:DeleteSecurityGroup", "ec2:DeleteDhcpOptions", "ec2:DeleteNatGateway", "ec2:DescribeVpcEndpoints", "ec2:DeleteVpc", "ec2:CreateSubnet", "ec2:DescribeSubnets" ], "Resource": "*" }, { "Sid": "ELB", "Effect": "Allow", "Action": [ "elasticloadbalancing:DeleteLoadBalancer", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DeleteTargetGroup" ], "Resource": "*" }, { "Sid": "IAMPassRole", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:*:iam::*:role/*-worker-role", "Condition": { "ForAnyValue:StringEqualsIfExists": { "iam:PassedToService": "ec2.amazonaws.com" } } }, { "Sid": "IAM", "Effect": "Allow", "Action": [ "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile", "iam:GetRole", "iam:UpdateAssumeRolePolicy", "iam:GetInstanceProfile", "iam:TagRole", "iam:RemoveRoleFromInstanceProfile", "iam:CreateRole", "iam:DeleteRole", "iam:PutRolePolicy", "iam:AddRoleToInstanceProfile", "iam:CreateOpenIDConnectProvider", "iam:ListOpenIDConnectProviders", "iam:DeleteRolePolicy", "iam:UpdateRole", "iam:DeleteOpenIDConnectProvider", "iam:GetRolePolicy" ], "Resource": "*" }, { "Sid": "Route53", "Effect": "Allow", "Action": [ "route53:ListHostedZonesByVPC", "route53:CreateHostedZone", "route53:ListHostedZones", "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets", "route53:DeleteHostedZone", "route53:AssociateVPCWithHostedZone", "route53:ListHostedZonesByName" ], "Resource": "*" }, { "Sid": "S3", "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets", "s3:ListBucket", "s3:DeleteObject", "s3:DeleteBucket" ], "Resource": "*" } ] } Attach the policy.json file to your role by running the following command: USD aws iam put-role-policy \ --role-name <role_name> \ 1 --policy-name <policy_name> \ 2 --policy-document file://policy.json 3 1 Replace <role_name> with the name of your role. 2 Replace <policy_name> with your policy name. 3 The policy.json file contains the permission policies for your role. Retrieve STS credentials in a JSON file named sts-creds.json by running the following command: USD aws sts get-session-token --output json > sts-creds.json Example sts-creds.json file { "Credentials": { "AccessKeyId": "ASIA1443CE0GN2ATHWJU", "SecretAccessKey": "XFLN7cZ5AP0d66KhyI4gd8Mu0UCQEDN9cfelW1", "SessionToken": "IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMiJHMEUCIDyipkM7oPKBHiGeI0pMnXst1gDLfs/TvfskXseKCbshAiEAnl1l/Html7Iq9AEIqf////KQburfkq4A3TuppHMr/9j1TgCj1z83SO261bHqlJUazKoy7vBFR/a6LHt55iMBqtKPEsIWjBgj/jSdRJI3j4Gyk1//luKDytcfF/tb9YrxDTPLrACS1lqAxSIFZ82I/jDhbDs=", "Expiration": "2025-05-16T04:19:32+00:00" } } 4.1.5. Enabling AWS PrivateLink for hosted control planes To provision hosted control planes on the Amazon Web Services (AWS) with PrivateLink, enable AWS PrivateLink for hosted control planes. Procedure Create an AWS credential secret for the HyperShift Operator and name it hypershift-operator-private-link-credentials . The secret must reside in the managed cluster namespace that is the namespace of the managed cluster being used as the management cluster. If you used local-cluster , create the secret in the local-cluster namespace. See the following table to confirm that the secret contains the required fields: Table 4.2. Required fields for the AWS secret Field name Description Optional or required region Region for use with Private Link Required aws-access-key-id The credential access key id. Required aws-secret-access-key The credential access key secret. Required To create an AWS secret, run the following command: USD oc create secret generic <secret_name> \ --from-literal=aws-access-key-id=<aws_access_key_id> \ --from-literal=aws-secret-access-key=<aws_secret_access_key> \ --from-literal=region=<region> -n local-cluster Note Disaster recovery backup for the secret is not automatically enabled. Run the following command to add the label that enables the hypershift-operator-private-link-credentials secret to be backed up for disaster recovery: USD oc label secret hypershift-operator-private-link-credentials \ -n local-cluster \ cluster.open-cluster-management.io/backup="" 4.1.6. Enabling external DNS for hosted control planes on AWS The control plane and the data plane are separate in hosted control planes. You can configure DNS in two independent areas: Ingress for workloads within the hosted cluster, such as the following domain: *.apps.service-consumer-domain.com . Ingress for service endpoints within the management cluster, such as API or OAuth endpoints through the service provider domain: *.service-provider-domain.com . The input for hostedCluster.spec.dns manages the ingress for workloads within the hosted cluster. The input for hostedCluster.spec.services.servicePublishingStrategy.route.hostname manages the ingress for service endpoints within the management cluster. External DNS creates name records for hosted cluster Services that specify a publishing type of LoadBalancer or Route and provide a hostname for that publishing type. For hosted clusters with Private or PublicAndPrivate endpoint access types, only the APIServer and OAuth services support hostnames. For Private hosted clusters, the DNS record resolves to a private IP address of a Virtual Private Cloud (VPC) endpoint in your VPC. A hosted control plane exposes the following services: APIServer OIDC You can expose these services by using the servicePublishingStrategy field in the HostedCluster specification. By default, for the LoadBalancer and Route types of servicePublishingStrategy , you can publish the service in one of the following ways: By using the hostname of the load balancer that is in the status of the Service with the LoadBalancer type. By using the status.host field of the Route resource. However, when you deploy hosted control planes in a managed service context, those methods can expose the ingress subdomain of the underlying management cluster and limit options for the management cluster lifecycle and disaster recovery. When a DNS indirection is layered on the LoadBalancer and Route publishing types, a managed service operator can publish all public hosted cluster services by using a service-level domain. This architecture allows remapping on the DNS name to a new LoadBalancer or Route and does not expose the ingress domain of the management cluster. Hosted control planes uses external DNS to achieve that indirection layer. You can deploy external-dns alongside the HyperShift Operator in the hypershift namespace of the management cluster. External DNS watches for Services or Routes that have the external-dns.alpha.kubernetes.io/hostname annotation. That annotation is used to create a DNS record that points to the Service , such as a record, or the Route , such as a CNAME record. You can use external DNS on cloud environments only. For the other environments, you need to manually configure DNS and services. For more information about external DNS, see external DNS . 4.1.6.1. Prerequisites Before you can set up external DNS for hosted control planes on Amazon Web Services (AWS), you must meet the following prerequisites: You created an external public domain. You have access to the AWS Route53 Management console. You enabled AWS PrivateLink for hosted control planes. 4.1.6.2. Setting up external DNS for hosted control planes You can provision hosted control planes with external DNS or service-level DNS. Create an Amazon Web Services (AWS) credential secret for the HyperShift Operator and name it hypershift-operator-external-dns-credentials in the local-cluster namespace. See the following table to verify that the secret has the required fields: Table 4.3. Required fields for the AWS secret Field name Description Optional or required provider The DNS provider that manages the service-level DNS zone. Required domain-filter The service-level domain. Required credentials The credential file that supports all external DNS types. Optional when you use AWS keys aws-access-key-id The credential access key id. Optional when you use the AWS DNS service aws-secret-access-key The credential access key secret. Optional when you use the AWS DNS service To create an AWS secret, run the following command: USD oc create secret generic <secret_name> \ --from-literal=provider=aws \ --from-literal=domain-filter=<domain_name> \ --from-file=credentials=<path_to_aws_credentials_file> -n local-cluster Note Disaster recovery backup for the secret is not automatically enabled. To back up the secret for disaster recovery, add the hypershift-operator-external-dns-credentials by entering the following command: USD oc label secret hypershift-operator-external-dns-credentials \ -n local-cluster \ cluster.open-cluster-management.io/backup="" 4.1.6.3. Creating the public DNS hosted zone The External DNS Operator uses the public DNS hosted zone to create your public hosted cluster. You can create the public DNS hosted zone to use as the external DNS domain-filter. Complete the following steps in the AWS Route 53 management console. Procedure In the Route 53 management console, click Create hosted zone . On the Hosted zone configuration page, type a domain name, verify that Publish hosted zone is selected as the type, and click Create hosted zone . After the zone is created, on the Records tab, note the values in the Value/Route traffic to column. In the main domain, create an NS record to redirect the DNS requests to the delegated zone. In the Value field, enter the values that you noted in the step. Click Create records . Verify that the DNS hosted zone is working by creating a test entry in the new subzone and testing it with a dig command, such as in the following example: USD dig +short test.user-dest-public.aws.kerberos.com Example output 192.168.1.1 To create a hosted cluster that sets the hostname for the LoadBalancer and Route services, enter the following command: USD hcp create cluster aws --name=<hosted_cluster_name> \ --endpoint-access=PublicAndPrivate \ --external-dns-domain=<public_hosted_zone> ... 1 1 Replace <public_hosted_zone> with the public hosted zone that you created. Example services block for the hosted cluster platform: aws: endpointAccess: PublicAndPrivate ... services: - service: APIServer servicePublishingStrategy: route: hostname: api-example.service-provider-domain.com type: Route - service: OAuthServer servicePublishingStrategy: route: hostname: oauth-example.service-provider-domain.com type: Route - service: Konnectivity servicePublishingStrategy: type: Route - service: Ignition servicePublishingStrategy: type: Route The Control Plane Operator creates the Services and Routes resources and annotates them with the external-dns.alpha.kubernetes.io/hostname annotation. For Services and Routes , the Control Plane Operator uses a value of the hostname parameter in the servicePublishingStrategy field for the service endpoints. To create the DNS records, you can use a mechanism, such as the external-dns deployment. You can configure service-level DNS indirection for public services only. You cannot set hostname for private services because they use the hypershift.local private zone. The following table shows when it is valid to set hostname for a service and endpoint combinations: Table 4.4. Service and endpoint combinations to set hostname Service Public PublicAndPrivate Private APIServer Y Y N OAuthServer Y Y N Konnectivity Y N N Ignition Y N N 4.1.6.4. Creating a hosted cluster by using the external DNS on AWS To create a hosted cluster by using the PublicAndPrivate or Public publishing strategy on Amazon Web Services (AWS), you must have the following artifacts configured in your management cluster: The public DNS hosted zone The External DNS Operator The HyperShift Operator You can deploy a hosted cluster, by using the hcp command-line interface (CLI). Procedure To access your management cluster, enter the following command: USD export KUBECONFIG=<path_to_management_cluster_kubeconfig> Verify that the External DNS Operator is running by entering the following command: USD oc get pod -n hypershift -lapp=external-dns Example output NAME READY STATUS RESTARTS AGE external-dns-7c89788c69-rn8gp 1/1 Running 0 40s To create a hosted cluster by using external DNS, enter the following command: USD hcp create cluster aws \ --role-arn <arn_role> \ 1 --instance-type <instance_type> \ 2 --region <region> \ 3 --auto-repair \ --generate-ssh \ --name <hosted_cluster_name> \ 4 --namespace clusters \ --base-domain <service_consumer_domain> \ 5 --node-pool-replicas <node_replica_count> \ 6 --pull-secret <path_to_your_pull_secret> \ 7 --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 8 --external-dns-domain=<service_provider_domain> \ 9 --endpoint-access=PublicAndPrivate 10 --sts-creds <path_to_sts_credential_file> 11 1 Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole . 2 Specify the instance type, for example, m6i.xlarge . 3 Specify the AWS region, for example, us-east-1 . 4 Specify your hosted cluster name, for example, my-external-aws . 5 Specify the public hosted zone that the service consumer owns, for example, service-consumer-domain.com . 6 Specify the node replica count, for example, 2 . 7 Specify the path to your pull secret file. 8 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . 9 Specify the public hosted zone that the service provider owns, for example, service-provider-domain.com . 10 Set as PublicAndPrivate . You can use external DNS with Public or PublicAndPrivate configurations only. 11 Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json . 4.1.7. Creating a hosted cluster on AWS You can create a hosted cluster on Amazon Web Services (AWS) by using the hcp command-line interface (CLI). By default for hosted control planes on Amazon Web Services (AWS), you use an AMD64 hosted cluster. However, you can enable hosted control planes to run on an ARM64 hosted cluster. For more information, see "Running hosted clusters on an ARM64 architecture". For compatible combinations of node pools and hosted clusters, see the following table: Table 4.5. Compatible architectures for node pools and hosted clusters Hosted cluster Node pools AMD64 AMD64 or ARM64 ARM64 ARM64 or AMD64 Prerequisites You have set up the hosted control plane CLI, hcp . You have enabled the local-cluster managed cluster as the management cluster. You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials. Procedure To create a hosted cluster on AWS, run the following command: USD hcp create cluster aws \ --name <hosted_cluster_name> \ 1 --infra-id <infra_id> \ 2 --base-domain <basedomain> \ 3 --sts-creds <path_to_sts_credential_file> \ 4 --pull-secret <path_to_pull_secret> \ 5 --region <region> \ 6 --generate-ssh \ --node-pool-replicas <node_pool_replica_count> \ 7 --namespace <hosted_cluster_namespace> \ 8 --role-arn <role_name> \ 9 --render-into <file_name>.yaml 10 1 Specify the name of your hosted cluster, for instance, example . 2 Specify your infrastructure name. You must provide the same value for <hosted_cluster_name> and <infra_id> . Otherwise the cluster might not appear correctly in the multicluster engine for Kubernetes Operator console. 3 Specify your base domain, for example, example.com . 4 Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json . 5 Specify the path to your pull secret, for example, /user/name/pullsecret . 6 Specify the AWS region name, for example, us-east-1 . 7 Specify the node pool replica count, for example, 3 . 8 By default, all HostedCluster and NodePool custom resources are created in the clusters namespace. You can use the --namespace <namespace> parameter, to create the HostedCluster and NodePool custom resources in a specific namespace. 9 Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole . 10 If you want to indicate whether the EC2 instance runs on shared or single tenant hardware, include this field. The --render-into flag renders Kubernetes resources into the YAML file that you specify in this field. Then, continue to the step to edit the YAML file. If you included the --render-into flag in the command, edit the specified YAML file. Edit the NodePool specification in the YAML file to indicate whether the EC2 instance should run on shared or single-tenant hardware, similar to the following example: Example YAML file apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool metadata: name: <nodepool_name> 1 spec: platform: aws: placement: tenancy: "default" 2 1 Specify the name of the NodePool resource. 2 Specify a valid value for tenancy: "default" , "dedicated" , or "host" . Use "default" when node pool instances run on shared hardware. Use "dedicated" when each node pool instance runs on single-tenant hardware. Use "host" when node pool instances run on your pre-allocated dedicated hosts. Verification Verify the status of your hosted cluster to check that the value of AVAILABLE is True . Run the following command: USD oc get hostedclusters -n <hosted_cluster_namespace> Get a list of your node pools by running the following command: USD oc get nodepools --namespace <hosted_cluster_namespace> Additional resources Running hosted clusters on an ARM64 architecture 4.1.7.1. Accessing a hosted cluster on AWS by using the kubeadmin credentials After creating a hosted cluster on Amazon Web Services (AWS), you can access a hosted cluster by getting the kubeconfig file, access secrets, and the kubeadmin credentials. The hosted cluster namespace contains hosted cluster resources and the access secrets. The hosted control plane runs in the hosted control plane namespace. The secret name formats are as follows: The kubeconfig secret: <hosted_cluster_namespace>-<name>-admin-kubeconfig . For example, clusters-hypershift-demo-admin-kubeconfig . The kubeadmin password secret: <hosted_cluster_namespace>-<name>-kubeadmin-password . For example, clusters-hypershift-demo-kubeadmin-password . Note The kubeadmin password secret is Base64-encoded and the kubeconfig secret contains a Base64-encoded kubeconfig configuration. You must decode the Base64-encoded kubeconfig configuration and save it into a <hosted_cluster_name>.kubeconfig file. Procedure Use your <hosted_cluster_name>.kubeconfig file that contains the decoded kubeconfig configuration to access the hosted cluster. Enter the following command: USD oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes You must decode the kubeadmin password secret to log in to the API server or the console of the hosted cluster. 4.1.7.2. Accessing a hosted cluster on AWS by using the hcp CLI You can access the hosted cluster by using the hcp command-line interface (CLI). Procedure Generate the kubeconfig file by entering the following command: USD hcp create kubeconfig --namespace <hosted_cluster_namespace> \ --name <hosted_cluster_name> > <hosted_cluster_name>.kubeconfig After you save the kubeconfig file, access the hosted cluster by entering the following command: USD oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes 4.1.8. Creating a hosted cluster in multiple zones on AWS You can create a hosted cluster in multiple zones on Amazon Web Services (AWS) by using the hcp command-line interface (CLI). Prerequisites You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials. Procedure Create a hosted cluster in multiple zones on AWS by running the following command: USD hcp create cluster aws \ --name <hosted_cluster_name> \ 1 --node-pool-replicas=<node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --role-arn <arn_role> \ 5 --region <region> \ 6 --zones <zones> \ 7 --sts-creds <path_to_sts_credential_file> 8 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the node pool replica count, for example, 2 . 3 Specify your base domain, for example, example.com . 4 Specify the path to your pull secret, for example, /user/name/pullsecret . 5 Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole . 6 Specify the AWS region name, for example, us-east-1 . 7 Specify availability zones within your AWS region, for example, us-east-1a , and us-east-1b . 8 Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json . For each specified zone, the following infrastructure is created: Public subnet Private subnet NAT gateway Private route table A public route table is shared across public subnets. One NodePool resource is created for each zone. The node pool name is suffixed by the zone name. The private subnet for zone is set in spec.platform.aws.subnet.id . 4.1.8.1. Creating a hosted cluster by providing AWS STS credentials When you create a hosted cluster by using the hcp create cluster aws command, you must provide an Amazon Web Services (AWS) account credentials that have permissions to create infrastructure resources for your hosted cluster. Infrastructure resources include the following examples: Virtual Private Cloud (VPC) Subnets Network address translation (NAT) gateways You can provide the AWS credentials by using the either of the following ways: The AWS Security Token Service (STS) credentials The AWS cloud provider secret from multicluster engine Operator Procedure To create a hosted cluster on AWS by providing AWS STS credentials, enter the following command: USD hcp create cluster aws \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --sts-creds <path_to_sts_credential_file> \ 5 --region <region> \ 6 --role-arn <arn_role> 7 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the node pool replica count, for example, 2 . 3 Specify your base domain, for example, example.com . 4 Specify the path to your pull secret, for example, /user/name/pullsecret . 5 Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json . 6 Specify the AWS region name, for example, us-east-1 . 7 Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole . 4.1.9. Running hosted clusters on an ARM64 architecture By default for hosted control planes on Amazon Web Services (AWS), you use an AMD64 hosted cluster. However, you can enable hosted control planes to run on an ARM64 hosted cluster. For compatible combinations of node pools and hosted clusters, see the following table: Table 4.6. Compatible architectures for node pools and hosted clusters Hosted cluster Node pools AMD64 AMD64 or ARM64 ARM64 ARM64 or AMD64 4.1.9.1. Creating a hosted cluster on an ARM64 OpenShift Container Platform cluster You can run a hosted cluster on an ARM64 OpenShift Container Platform cluster for Amazon Web Services (AWS) by overriding the default release image with a multi-architecture release image. If you do not use a multi-architecture release image, the compute nodes in the node pool are not created and reconciliation of the node pool stops until you either use a multi-architecture release image in the hosted cluster or update the NodePool custom resource based on the release image. Prerequisites You must have an OpenShift Container Platform cluster with a 64-bit ARM infrastructure that is installed on AWS. For more information, see Create an OpenShift Container Platform Cluster: AWS (ARM) . You must create an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials. For more information, see "Creating an AWS IAM role and STS credentials". Procedure Create a hosted cluster on an ARM64 OpenShift Container Platform cluster by entering the following command: USD hcp create cluster aws \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --sts-creds <path_to_sts_credential_file> \ 5 --region <region> \ 6 --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 7 --role-arn <role_name> 8 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the node pool replica count, for example, 3 . 3 Specify your base domain, for example, example.com . 4 Specify the path to your pull secret, for example, /user/name/pullsecret . 5 Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json . 6 Specify the AWS region name, for example, us-east-1 . 7 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see "Extracting the OpenShift Container Platform release image digest". 8 Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole . 4.1.9.2. Creating an ARM or AMD NodePool object on AWS hosted clusters You can schedule application workloads that is the NodePool objects on 64-bit ARM and AMD from the same hosted control plane. You can define the arch field in the NodePool specification to set the required processor architecture for the NodePool object. The valid values for the arch field are as follows: arm64 amd64 Prerequisites You must have a multi-architecture image for the HostedCluster custom resource to use. You can access multi-architecture nightly images . Procedure Add an ARM or AMD NodePool object to the hosted cluster on AWS by running the following command: USD hcp create nodepool aws \ --cluster-name <hosted_cluster_name> \ 1 --name <node_pool_name> \ 2 --node-count <node_pool_replica_count> \ 3 --arch <architecture> 4 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the node pool name. 3 Specify the node pool replica count, for example, 3 . 4 Specify the architecture type, such as arm64 or amd64 . If you do not specify a value for the --arch flag, the amd64 value is used by default. Additional resources Extracting the OpenShift Container Platform release image digest 4.1.10. Creating a private hosted cluster on AWS After you enable the local-cluster as the hosting cluster, you can deploy a hosted cluster or a private hosted cluster on Amazon Web Services (AWS). By default, hosted clusters are publicly accessible through public DNS and the default router for the management cluster. For private clusters on AWS, all communication with the hosted cluster occurs over AWS PrivateLink. Prerequisites You enabled AWS PrivateLink. For more information, see "Enabling AWS PrivateLink". You created an AWS Identity and Access Management (IAM) role and AWS Security Token Service (STS) credentials. For more information, see "Creating an AWS IAM role and STS credentials" and "Identity and Access Management (IAM) permissions". You configured a bastion instance on AWS . Procedure Create a private hosted cluster on AWS by entering the following command: USD hcp create cluster aws \ --name <hosted_cluster_name> \ 1 --node-pool-replicas=<node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --sts-creds <path_to_sts_credential_file> \ 5 --region <region> \ 6 --endpoint-access Private \ 7 --role-arn <role_name> 8 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the node pool replica count, for example, 3 . 3 Specify your base domain, for example, example.com . 4 Specify the path to your pull secret, for example, /user/name/pullsecret . 5 Specify the path to your AWS STS credentials file, for example, /home/user/sts-creds/sts-creds.json . 6 Specify the AWS region name, for example, us-east-1 . 7 Defines whether a cluster is public or private. 8 Specify the Amazon Resource Name (ARN), for example, arn:aws:iam::820196288204:role/myrole . For more information about ARN roles, see "Identity and Access Management (IAM) permissions". The following API endpoints for the hosted cluster are accessible through a private DNS zone: api.<hosted_cluster_name>.hypershift.local *.apps.<hosted_cluster_name>.hypershift.local 4.1.10.1. Accessing a private management cluster on AWS Additional resources You can access your private management cluster by using the command-line interface (CLI). Procedure Find the private IPs of nodes by entering the following command: USD aws ec2 describe-instances \ --filter="Name=tag:kubernetes.io/cluster/<infra_id>,Values=owned" \ | jq '.Reservations[] | .Instances[] | select(.PublicDnsName=="") \ | .PrivateIpAddress' Create a kubeconfig file for the hosted cluster that you can copy to a node by entering the following command: USD hcp create kubeconfig > <hosted_cluster_kubeconfig> To SSH into one of the nodes through the bastion, enter the following command: USD ssh -o ProxyCommand="ssh ec2-user@<bastion_ip> \ -W %h:%p" core@<node_ip> From the SSH shell, copy the kubeconfig file contents to a file on the node by entering the following command: USD mv <path_to_kubeconfig_file> <new_file_name> Export the kubeconfig file by entering the following command: USD export KUBECONFIG=<path_to_kubeconfig_file> Observe the hosted cluster status by entering the following command: USD oc get clusteroperators clusterversion 4.2. Deploying hosted control planes on bare metal You can deploy hosted control planes by configuring a cluster to function as a management cluster. The management cluster is the OpenShift Container Platform cluster where the control planes are hosted. In some contexts, the management cluster is also known as the hosting cluster. Note The management cluster is not the same thing as the managed cluster. A managed cluster is a cluster that the hub cluster manages. The hosted control planes feature is enabled by default. The multicluster engine Operator supports only the default local-cluster , which is a hub cluster that is managed, and the hub cluster as the management cluster. If you have Red Hat Advanced Cluster Management installed, you can use the managed hub cluster, also known as the local-cluster , as the management cluster. A hosted cluster is an OpenShift Container Platform cluster with its API endpoint and control plane that are hosted on the management cluster. The hosted cluster includes the control plane and its corresponding data plane. You can use the multicluster engine Operator console or the hosted control plane command line interface, hcp , to create a hosted cluster. The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see Disabling the automatic import of hosted clusters into multicluster engine Operator . 4.2.1. Preparing to deploy hosted control planes on bare metal As you prepare to deploy hosted control planes on bare metal, consider the following information: Run the management cluster and workers on the same platform for hosted control planes. All bare metal hosts require a manual start with a Discovery Image ISO that the central infrastructure management provides. You can start the hosts manually or through automation by using Cluster-Baremetal-Operator . After each host starts, it runs an Agent process to discover the host details and complete the installation. An Agent custom resource represents each host. When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control plane etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using logical volume manager storage". 4.2.1.1. Prerequisites to configure a management cluster You need the multicluster engine for Kubernetes Operator 2.2 and later installed on an OpenShift Container Platform cluster. You can install multicluster engine Operator as an Operator from the OpenShift Container Platform OperatorHub. The multicluster engine Operator must have at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported in multicluster engine Operator 2.2 and later. For more information about the local-cluster , see Advanced configuration in the Red Hat Advanced Cluster Management documentation. You can check the status of your hub cluster by running the following command: USD oc get managedclusters local-cluster You must add the topology.kubernetes.io/zone label to your bare-metal hosts on your management cluster. Ensure that each host has a unique value for topology.kubernetes.io/zone . Otherwise, all of the hosted control plane pods are scheduled on a single node, causing a single point of failure. To provision hosted control planes on bare metal, you can use the Agent platform. The Agent platform uses the central infrastructure management service to add worker nodes to a hosted cluster. For more information, see Enabling the central infrastructure management service . You need to install the hosted control plane command line interface. Additional resources Advanced configuration Enabling the central infrastructure management service 4.2.1.2. Bare metal firewall, port, and service requirements You must meet the firewall, port, and service requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters. Note Services run on their default ports. However, if you use the NodePort publishing strategy, services run on the port that is assigned by the NodePort service. Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary. For production deployments, use a load balancer to simplify access through a single IP address. If your hub cluster has a proxy configuration, ensure that it can reach the hosted cluster API endpoint by adding all hosted cluster API endpoints to the noProxy field on the Proxy object. For more information, see "Configuring the cluster-wide proxy". A hosted control plane exposes the following services on bare metal: APIServer The APIServer service runs on port 6443 by default and requires ingress access for communication between the control plane components. If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses. OAuthServer The OAuthServer service runs on port 443 by default when you use the route and ingress to expose the service. If you use the NodePort publishing strategy, use a firewall rule for the OAuthServer service. Konnectivity The Konnectivity service runs on port 443 by default when you use the route and ingress to expose the service. The Konnectivity agent establishes a reverse tunnel to allow the control plane to access the network for the hosted cluster. The agent uses egress to connect to the Konnectivity server. The server is exposed by using either a route on port 443 or a manually assigned NodePort . If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443. If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes. Ignition The Ignition service runs on port 443 by default when you use the route and ingress to expose the service. If you use the NodePort publishing strategy, use a firewall rule for the Ignition service. You do not need the following services on bare metal: OVNSbDb OIDC Additional resources Configuring the cluster-wide proxy 4.2.1.3. Bare metal infrastructure requirements The Agent platform does not create any infrastructure, but it does have the following requirements for infrastructure: Agents: An Agent represents a host that is booted with a discovery image and is ready to be provisioned as an OpenShift Container Platform node. DNS: The API and ingress endpoints must be routable. Additional resources Recommended etcd practices Persistent storage using LVM Storage Disabling the automatic import of hosted clusters into multicluster engine Operator Enabling or disabling the hosted control planes feature Configuring Ansible Automation Platform jobs to run on hosted clusters 4.2.2. DNS configurations on bare metal The API Server for the hosted cluster is exposed as a NodePort service. A DNS entry must exist for api.<hosted_cluster_name>.<base_domain> that points to destination where the API Server can be reached. The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane. The entry can also point to a load balancer that is deployed to redirect incoming traffic to the ingress pods. Example DNS configuration api.example.krnl.es. IN A 192.168.122.20 api.example.krnl.es. IN A 192.168.122.21 api.example.krnl.es. IN A 192.168.122.22 api-int.example.krnl.es. IN A 192.168.122.20 api-int.example.krnl.es. IN A 192.168.122.21 api-int.example.krnl.es. IN A 192.168.122.22 `*`.apps.example.krnl.es. IN A 192.168.122.23 If you are configuring DNS for a disconnected environment on an IPv6 network, the configuration looks like the following example. Example DNS configuration for an IPv6 network api.example.krnl.es. IN A 2620:52:0:1306::5 api.example.krnl.es. IN A 2620:52:0:1306::6 api.example.krnl.es. IN A 2620:52:0:1306::7 api-int.example.krnl.es. IN A 2620:52:0:1306::5 api-int.example.krnl.es. IN A 2620:52:0:1306::6 api-int.example.krnl.es. IN A 2620:52:0:1306::7 `*`.apps.example.krnl.es. IN A 2620:52:0:1306::10 If you are configuring DNS for a disconnected environment on a dual stack network, be sure to include DNS entries for both IPv4 and IPv6. Example DNS configuration for a dual stack network host-record=api-int.hub-dual.dns.base.domain.name,192.168.126.10 host-record=api.hub-dual.dns.base.domain.name,192.168.126.10 address=/apps.hub-dual.dns.base.domain.name/192.168.126.11 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,192.168.126.20 dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,192.168.126.21 dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,192.168.126.22 dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,192.168.126.25 dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,192.168.126.26 host-record=api-int.hub-dual.dns.base.domain.name,2620:52:0:1306::2 host-record=api.hub-dual.dns.base.domain.name,2620:52:0:1306::2 address=/apps.hub-dual.dns.base.domain.name/2620:52:0:1306::3 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5] dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,[2620:52:0:1306::6] dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,[2620:52:0:1306::7] dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,[2620:52:0:1306::8] dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,[2620:52:0:1306::9] 4.2.3. Creating a hosted cluster on bare metal When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one. As you create a hosted cluster, keep the following guidelines in mind: Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it. Do not use clusters as a hosted cluster name. A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster. The most common service publishing strategy is to expose services through a load balancer. That strategy is the preferred method for exposing the Kubernetes API server. If you create a hosted cluster by using the web console or by using Red Hat Advanced Cluster Management, to set a publishing strategy for a service besides the Kubernetes API server, you must manually specify the servicePublishingStrategy information in the HostedCluster custom resource. Procedure Create the hosted control plane namespace by entering the following command: USD oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters . Replace <hosted_cluster_name> with your hosted cluster name. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command: USD hcp create cluster agent \ --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_public_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 10 --node-pool-replicas <node_pool_replica_count> 11 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the path to your pull secret, for example, /user/name/pullsecret . 3 Specify your hosted control plane namespace, for example, clusters-example . Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command. 4 Specify your base domain, for example, krnl.es . 5 The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster. 6 Specify the etcd storage class name, for example, lvm-storageclass . 7 Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub . 8 Specify your hosted cluster namespace. 9 Specify the availability policy for the hosted control plane components. Supported options are SingleReplica and HighlyAvailable . The default value is HighlyAvailable . 10 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest . 11 Specify the node pool replica count, for example, 3 . You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, no node pools are created. After a few moments, verify that your hosted control plane pods are up and running by entering the following command: USD oc -n <hosted_control_plane_namespace> get pods Example output NAME READY STATUS RESTARTS AGE capi-provider-7dcf5fc4c4-nr9sq 1/1 Running 0 4m32s catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s certified-operators-catalog-884c756c4-zdt64 1/1 Running 0 2m51s cluster-api-f75d86f8c-56wfz 1/1 Running 0 4m32s Additional resources Manually importing a hosted cluster 4.2.3.1. Creating a hosted cluster on bare metal by using the console To create a hosted cluster by using the console, complete the following steps. Procedure Open the OpenShift Container Platform web console and log in by entering your administrator credentials. For instructions to open the console, see Accessing the web console . In the console header, ensure that All Clusters is selected. Click Infrastructure Clusters . Click Create cluster Host inventory Hosted control plane . The Create cluster page is displayed. On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation. Note As you enter details about the cluster, you might find the following tips useful: If you want to use predefined values to automatically populate fields in the console, you can create a host inventory credential. For more information, see Creating a credential for an on-premises environment . On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a host inventory credential, the pull secret is automatically populated. On the Node pools page, the namespace contains the hosts for the node pool. If you created a host inventory by using the console, the console creates a dedicated namespace. On the Networking page, you select an API server publishing strategy. The API server for the hosted cluster can be exposed either by using an existing load balancer or as a service of the NodePort type. A DNS entry must exist for the api.<hosted_cluster_name>.<base_domain> setting that points to the destination where the API server can be reached. This entry can be a record that points to one of the nodes in the management cluster or a record that points to a load balancer that redirects incoming traffic to the Ingress pods. Review your entries and click Create . The Hosted cluster view is displayed. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, then click the cluster name. Wait until the control plane components are ready. This process can take a few minutes. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster. steps To access the web console, see Accessing the web console . 4.2.3.2. Creating a hosted cluster on bare metal by using a mirror registry You can use a mirror registry to create a hosted cluster on bare metal by specifying the --image-content-sources flag in the hcp create cluster command. Procedure Create a YAML file to define Image Content Source Policies (ICSP). See the following example: - mirrors: - brew.registry.redhat.io source: registry.redhat.io - mirrors: - brew.registry.redhat.io source: registry.stage.redhat.io - mirrors: - brew.registry.redhat.io source: registry-proxy.engineering.redhat.com Save the file as icsp.yaml . This file contains your mirror registries. To create a hosted cluster by using your mirror registries, run the following command: USD hcp create cluster agent \ --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --image-content-sources icsp.yaml \ 6 --ssh-key <path_to_ssh_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the path to your pull secret, for example, /user/name/pullsecret . 3 Specify your hosted control plane namespace, for example, clusters-example . Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command. 4 Specify your base domain, for example, krnl.es . 5 The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster. 6 Specify the icsp.yaml file that defines ICSP and your mirror registries. 7 Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub . 8 Specify your hosted cluster namespace. 9 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest . steps To create credentials that you can reuse when you create a hosted cluster with the console, see Creating a credential for an on-premises environment . To access a hosted cluster, see Accessing the hosted cluster . To add hosts to the host inventory by using the Discovery Image, see Adding hosts to the host inventory by using the Discovery Image . To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest . 4.2.4. Verifying hosted cluster creation After the deployment process is complete, you can verify that the hosted cluster was created successfully. Follow these steps a few minutes after you create the hosted cluster. Procedure Obtain the kubeconfig for your new hosted cluster by entering the extract command: USD oc extract -n <hosted-control-plane-namespace> secret/admin-kubeconfig \ --to=- > kubeconfig-<hosted-cluster-name> Use the kubeconfig to view the cluster Operators of the hosted cluster. Enter the following command: USD oc get co --kubeconfig=kubeconfig-<hosted-cluster-name> Example output You can also view the running pods on your hosted cluster by entering the following command: USD oc get pods -A --kubeconfig=kubeconfig-<hosted-cluster-name> Example output 4.3. Deploying hosted control planes on OpenShift Virtualization With hosted control planes and OpenShift Virtualization, you can create OpenShift Container Platform clusters with worker nodes that are hosted by KubeVirt virtual machines. Hosted control planes on OpenShift Virtualization provides several benefits: Enhances resource usage by packing hosted control planes and hosted clusters in the same underlying bare metal infrastructure Separates hosted control planes and hosted clusters to provide strong isolation Reduces cluster provision time by eliminating the bare metal node bootstrapping process Manages many releases under the same base OpenShift Container Platform cluster The hosted control planes feature is enabled by default. You can use the hosted control plane command line interface, hcp , to create an OpenShift Container Platform hosted cluster. The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see "Disabling the automatic import of hosted clusters into multicluster engine Operator". Additional resources Disabling the automatic import of hosted clusters into multicluster engine Operator Enabling or disabling the hosted control planes feature Configuring Ansible Automation Platform jobs to run on hosted clusters 4.3.1. Requirements to deploy hosted control planes on OpenShift Virtualization As you prepare to deploy hosted control planes on OpenShift Virtualization, consider the following information: Run the management cluster on bare metal. Each hosted cluster must have a cluster-wide unique name. Do not use clusters as a hosted cluster name. A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster. When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control plane etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using Logical Volume Manager storage". Additional resources Recommended etcd practices Persistent storage using Logical Volume Manager Storage 4.3.1.1. Prerequisites You must meet the following prerequisites to create an OpenShift Container Platform cluster on OpenShift Virtualization: You have administrator access to an OpenShift Container Platform cluster, version 4.14 or later, specified in the KUBECONFIG environment variable. The OpenShift Container Platform management cluster has wildcard DNS routes enabled, as shown in the following DNS: USD oc patch ingresscontroller -n openshift-ingress-operator default \ --type=json \ -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]' The OpenShift Container Platform management cluster has OpenShift Virtualization, version 4.14 or later, installed on it. For more information, see "Installing OpenShift Virtualization using the web console". The OpenShift Container Platform management cluster is on-premise bare metal. The OpenShift Container Platform management cluster is configured with OVNKubernetes as the default pod network CNI. The OpenShift Container Platform management cluster has a default storage class. For more information, see "Postinstallation storage configuration". The following example shows how to set a default storage class: USD oc patch storageclass ocs-storagecluster-ceph-rbd \ -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' You have a valid pull secret file for the quay.io/openshift-release-dev repository. For more information, see "Install OpenShift on any x86_64 platform with user-provisioned infrastructure". You have installed the hosted control plane command line interface. You have configured a load balancer. For more information, see "Configuring MetalLB". For optimal network performance, you are using a network maximum transmission unit (MTU) of 9000 or greater on the OpenShift Container Platform cluster that hosts the KubeVirt virtual machines. If you use a lower MTU setting, network latency and the throughput of the hosted pods are affected. Enable multiqueue on node pools only when the MTU is 9000 or greater. The multicluster engine Operator has at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported. For more information about the local-cluster , see "Advanced configuration" in the multicluster engine Operator documentation. You can check the status of your hub cluster by running the following command: USD oc get managedclusters local-cluster On the OpenShift Container Platform cluster that hosts the OpenShift Virtualization virtual machines, you are using a ReadWriteMany (RWX) storage class so that live migration can be enabled. Additional resources Installing OpenShift Virtualization using the web console Postinstallation storage configuration Install OpenShift on any x86_64 platform with user-provisioned infrastructure Configuring MetalLB Advanced configuration 4.3.1.2. Firewall and port requirements Ensure that you meet the firewall and port requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters: The kube-apiserver service runs on port 6443 by default and requires ingress access for communication between the control plane components. If you use the NodePort publishing strategy, ensure that the node port that is assigned to the kube-apiserver service is exposed. If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses. If you use the NodePort publishing strategy, use a firewall rule for the ignition-server and Oauth-server settings. The konnectivity agent, which establishes a reverse tunnel to allow bi-directional communication on the hosted cluster, requires egress access to the cluster API server address on port 6443. With that egress access, the agent can reach the kube-apiserver service. If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443. If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes. If you change the default port of 6443, adjust the rules to reflect that change. Ensure that you open any ports that are required by the workloads that run in the clusters. Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary. For production deployments, use a load balancer to simplify access through a single IP address. 4.3.2. Live migration for compute nodes While the management cluster for hosted cluster virtual machines (VMs) is undergoing updates or maintenance, the hosted cluster VMs can be automatically live migrated to prevent disrupting hosted cluster workloads. As a result, the management cluster can be updated without affecting the availability and operation of the KubeVirt platform hosted clusters. Important The live migration of KubeVirt VMs is enabled by default provided that the VMs use ReadWriteMany (RWX) storage for both the root volume and the storage classes that are mapped to the kubevirt-csi CSI provider. You can verify that the VMs in a node pool are capable of live migration by checking the KubeVirtNodesLiveMigratable condition in the status section of a NodePool object. In the following example, the VMs cannot be live migrated because RWX storage is not used. Example configuration where VMs cannot be live migrated - lastTransitionTime: "2024-10-08T15:38:19Z" message: | 3 of 3 machines are not live migratable Machine user-np-ngst4-gw2hz: DisksNotLiveMigratable: user-np-ngst4-gw2hz is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-gw2hz-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) Machine user-np-ngst4-npq7x: DisksNotLiveMigratable: user-np-ngst4-npq7x is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-npq7x-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) Machine user-np-ngst4-q5nkb: DisksNotLiveMigratable: user-np-ngst4-q5nkb is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-q5nkb-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) observedGeneration: 1 reason: DisksNotLiveMigratable status: "False" type: KubeVirtNodesLiveMigratable In the example, the VMs meet the requirements to be live migrated. Example configuration where VMs can be live migrated - lastTransitionTime: "2024-10-08T15:38:19Z" message: "All is well" observedGeneration: 1 reason: AsExpected status: "True" type: KubeVirtNodesLiveMigratable While live migration can protect VMs from disruption in normal circumstances, events such as infrastructure node failure can result in a hard restart of any VMs that are hosted on the failed node. For live migration to be successful, the source node that a VM is hosted on must be working correctly. When the VMs in a node pool cannot be live migrated, workload disruption might occur on the hosted cluster during maintenance on the management cluster. By default, the hosted control planes controllers try to drain the workloads that are hosted on KubeVirt VMs that cannot be live migrated before the VMs are stopped. Draining the hosted cluster nodes before stopping the VMs allows pod disruption budgets to protect workload availability within the hosted cluster. 4.3.3. Creating a hosted cluster with the KubeVirt platform With OpenShift Container Platform 4.14 and later, you can create a cluster with KubeVirt, to include creating with an external infrastructure. 4.3.3.1. Creating a hosted cluster with the KubeVirt platform by using the CLI To create a hosted cluster, you can use the hosted control plane command-line interface, hcp . Procedure Create a hosted cluster with the KubeVirt platform by entering the following command: USD hcp create cluster kubevirt \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <node_pool_replica_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <value_for_memory> \ 4 --cores <value_for_cpu> \ 5 --etcd-storage-class=<etcd_storage_class> 6 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the node pool replica count, for example, 3 . You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, no node pools are created. 3 Specify the path to your pull secret, for example, /user/name/pullsecret . 4 Specify a value for memory, for example, 6Gi . 5 Specify a value for CPU, for example, 2 . 6 Specify the etcd storage class name, for example, lvm-storageclass . Note You can use the --release-image flag to set up the hosted cluster with a specific OpenShift Container Platform release. A default node pool is created for the cluster with two virtual machine worker replicas according to the --node-pool-replicas flag. After a few moments, verify that the hosted control plane pods are running by entering the following command: USD oc -n clusters-<hosted-cluster-name> get pods Example output NAME READY STATUS RESTARTS AGE capi-provider-5cc7b74f47-n5gkr 1/1 Running 0 3m catalog-operator-5f799567b7-fd6jw 2/2 Running 0 69s certified-operators-catalog-784b9899f9-mrp6p 1/1 Running 0 66s cluster-api-6bbc867966-l4dwl 1/1 Running 0 66s . . . redhat-operators-catalog-9d5fd4d44-z8qqk 1/1 Running 0 66s A hosted cluster that has worker nodes that are backed by KubeVirt virtual machines typically takes 10-15 minutes to be fully provisioned. To check the status of the hosted cluster, see the corresponding HostedCluster resource by entering the following command: USD oc get --namespace clusters hostedclusters See the following example output, which illustrates a fully provisioned HostedCluster object: Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use. 4.3.3.2. Creating a hosted cluster with the KubeVirt platform by using external infrastructure By default, the HyperShift Operator hosts both the control plane pods of the hosted cluster and the KubeVirt worker VMs within the same cluster. With the external infrastructure feature, you can place the worker node VMs on a separate cluster from the control plane pods. The management cluster is the OpenShift Container Platform cluster that runs the HyperShift Operator and hosts the control plane pods for a hosted cluster. The infrastructure cluster is the OpenShift Container Platform cluster that runs the KubeVirt worker VMs for a hosted cluster. By default, the management cluster also acts as the infrastructure cluster that hosts VMs. However, for external infrastructure, the management and infrastructure clusters are different. Prerequisites You must have a namespace on the external infrastructure cluster for the KubeVirt nodes to be hosted in. You must have a kubeconfig file for the external infrastructure cluster. Procedure You can create a hosted cluster by using the hcp command line interface. To place the KubeVirt worker VMs on the infrastructure cluster, use the --infra-kubeconfig-file and --infra-namespace arguments, as shown in the following example: USD hcp create cluster kubevirt \ --name <hosted-cluster-name> \ 1 --node-pool-replicas <worker-count> \ 2 --pull-secret <path-to-pull-secret> \ 3 --memory <value-for-memory> \ 4 --cores <value-for-cpu> \ 5 --infra-namespace=<hosted-cluster-namespace>-<hosted-cluster-name> \ 6 --infra-kubeconfig-file=<path-to-external-infra-kubeconfig> 7 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the worker count, for example, 2 . 3 Specify the path to your pull secret, for example, /user/name/pullsecret . 4 Specify a value for memory, for example, 6Gi . 5 Specify a value for CPU, for example, 2 . 6 Specify the infrastructure namespace, for example, clusters-example . 7 Specify the path to your kubeconfig file for the infrastructure cluster, for example, /user/name/external-infra-kubeconfig . After you enter that command, the control plane pods are hosted on the management cluster that the HyperShift Operator runs on, and the KubeVirt VMs are hosted on a separate infrastructure cluster. 4.3.3.3. Creating a hosted cluster by using the console To create a hosted cluster with the KubeVirt platform by using the console, complete the following steps. Procedure Open the OpenShift Container Platform web console and log in by entering your administrator credentials. In the console header, ensure that All Clusters is selected. Click Infrastructure > Clusters . Click Create cluster > Red Hat OpenShift Virtualization > Hosted . On the Create cluster page, follow the prompts to enter details about the cluster and node pools. Note If you want to use predefined values to automatically populate fields in the console, you can create a OpenShift Virtualization credential. For more information, see Creating a credential for an on-premises environment . On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a OpenShift Virtualization credential, the pull secret is automatically populated. Review your entries and click Create . The Hosted cluster view is displayed. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, and click the cluster name. Wait until the control plane components are ready. This process can take a few minutes. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster. Additional resources To create credentials that you can reuse when you create a hosted cluster with the console, see Creating a credential for an on-premises environment . To access the hosted cluster, see Accessing the hosted cluster . 4.3.4. Configuring the default ingress and DNS for hosted control planes on OpenShift Virtualization Every OpenShift Container Platform cluster includes a default application Ingress Controller, which must have an wildcard DNS record associated with it. By default, hosted clusters that are created by using the HyperShift KubeVirt provider automatically become a subdomain of the OpenShift Container Platform cluster that the KubeVirt virtual machines run on. For example, your OpenShift Container Platform cluster might have the following default ingress DNS entry: *.apps.mgmt-cluster.example.com As a result, a KubeVirt hosted cluster that is named guest and that runs on that underlying OpenShift Container Platform cluster has the following default ingress: *.apps.guest.apps.mgmt-cluster.example.com Procedure For the default ingress DNS to work properly, the cluster that hosts the KubeVirt virtual machines must allow wildcard DNS routes. You can configure this behavior by entering the following command: USD oc patch ingresscontroller -n openshift-ingress-operator default \ --type=json \ -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]' Note When you use the default hosted cluster ingress, connectivity is limited to HTTPS traffic over port 443. Plain HTTP traffic over port 80 is rejected. This limitation applies to only the default ingress behavior. 4.3.5. Customizing ingress and DNS behavior If you do not want to use the default ingress and DNS behavior, you can configure a KubeVirt hosted cluster with a unique base domain at creation time. This option requires manual configuration steps during creation and involves three main steps: cluster creation, load balancer creation, and wildcard DNS configuration. 4.3.5.1. Deploying a hosted cluster that specifies the base domain To create a hosted cluster that specifies a base domain, complete the following steps. Procedure Enter the following command: USD hcp create cluster kubevirt \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <value_for_memory> \ 4 --cores <value_for_cpu> \ 5 --base-domain <basedomain> 6 1 Specify the name of your hosted cluster. 2 Specify the worker count, for example, 2 . 3 Specify the path to your pull secret, for example, /user/name/pullsecret . 4 Specify a value for memory, for example, 6Gi . 5 Specify a value for CPU, for example, 2 . 6 Specify the base domain, for example, hypershift.lab . As a result, the hosted cluster has an ingress wildcard that is configured for the cluster name and the base domain, for example, .apps.example.hypershift.lab . The hosted cluster remains in Partial status because after you create a hosted cluster with unique base domain, you must configure the required DNS records and load balancer. View the status of your hosted cluster by entering the following command: USD oc get --namespace clusters hostedclusters Example output NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE example example-admin-kubeconfig Partial True False The hosted control plane is available Access the cluster by entering the following commands: USD hcp create kubeconfig --name <hosted_cluster_name> \ > <hosted_cluster_name>-kubeconfig USD oc --kubeconfig <hosted_cluster_name>-kubeconfig get co Example output NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console <4.x.0> False False False 30m RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.example.hypershift.lab): Get "https://console-openshift-console.apps.example.hypershift.lab": dial tcp: lookup console-openshift-console.apps.example.hypershift.lab on 172.31.0.10:53: no such host ingress <4.x.0> True False True 28m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing) Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use. steps To fix the errors in the output, complete the steps in "Setting up the load balancer" and "Setting up a wildcard DNS". Note If your hosted cluster is on bare metal, you might need MetalLB to set up load balancer services. For more information, see "Configuring MetalLB". 4.3.5.2. Setting up the load balancer Set up the load balancer service that routes ingress traffic to the KubeVirt VMs and assigns a wildcard DNS entry to the load balancer IP address. Procedure A NodePort service that exposes the hosted cluster ingress already exists. You can export the node ports and create the load balancer service that targets those ports. Get the HTTP node port by entering the following command: USD oc --kubeconfig <hosted_cluster_name>-kubeconfig get services \ -n openshift-ingress router-nodeport-default \ -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}' Note the HTTP node port value to use in the step. Get the HTTPS node port by entering the following command: USD oc --kubeconfig <hosted_cluster_name>-kubeconfig get services \ -n openshift-ingress router-nodeport-default \ -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}' Note the HTTPS node port value to use in the step. Create the load balancer service by entering the following command: oc apply -f - apiVersion: v1 kind: Service metadata: labels: app: <hosted_cluster_name> name: <hosted_cluster_name>-apps namespace: clusters-<hosted_cluster_name> spec: ports: - name: https-443 port: 443 protocol: TCP targetPort: <https_node_port> 1 - name: http-80 port: 80 protocol: TCP targetPort: <http-node-port> 2 selector: kubevirt.io: virt-launcher type: LoadBalancer 1 Specify the HTTPS node port value that you noted in the step. 2 Specify the HTTP node port value that you noted in the step. 4.3.5.3. Setting up a wildcard DNS Set up a wildcard DNS record or CNAME that references the external IP of the load balancer service. Procedure Get the external IP address by entering the following command: USD oc -n clusters-<hosted_cluster_name> get service <hosted-cluster-name>-apps \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}' Example output 192.168.20.30 Configure a wildcard DNS entry that references the external IP address. View the following example DNS entry: *.apps.<hosted_cluster_name\>.<base_domain\>. The DNS entry must be able to route inside and outside of the cluster. DNS resolutions example dig +short test.apps.example.hypershift.lab 192.168.20.30 Check that hosted cluster status has moved from Partial to Completed by entering the following command: USD oc get --namespace clusters hostedclusters Example output NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE example <4.x.0> example-admin-kubeconfig Completed True False The hosted control plane is available Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use. 4.3.6. Configuring MetalLB You must install the MetalLB Operator before you configure MetalLB. Procedure Complete the following steps to configure MetalLB on your hosted cluster: Create a MetalLB resource by saving the following sample YAML content in the configure-metallb.yaml file: apiVersion: metallb.io/v1beta1 kind: MetalLB metadata: name: metallb namespace: metallb-system Apply the YAML content by entering the following command: USD oc apply -f configure-metallb.yaml Example output metallb.metallb.io/metallb created Create a IPAddressPool resource by saving the following sample YAML content in the create-ip-address-pool.yaml file: apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: metallb namespace: metallb-system spec: addresses: - 192.168.216.32-192.168.216.122 1 1 Create an address pool with an available range of IP addresses within the node network. Replace the IP address range with an unused pool of available IP addresses in your network. Apply the YAML content by entering the following command: USD oc apply -f create-ip-address-pool.yaml Example output ipaddresspool.metallb.io/metallb created Create a L2Advertisement resource by saving the following sample YAML content in the l2advertisement.yaml file: apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: l2advertisement namespace: metallb-system spec: ipAddressPools: - metallb Apply the YAML content by entering the following command: USD oc apply -f l2advertisement.yaml Example output l2advertisement.metallb.io/metallb created Additional resources For more information about MetalLB, see Installing the MetalLB Operator . 4.3.7. Configuring additional networks, guaranteed CPUs, and VM scheduling for node pools If you need to configure additional networks for node pools, request a guaranteed CPU access for Virtual Machines (VMs), or manage scheduling of KubeVirt VMs, see the following procedures. 4.3.7.1. Adding multiple networks to a node pool By default, nodes generated by a node pool are attached to the pod network. You can attach additional networks to the nodes by using Multus and NetworkAttachmentDefinitions. Procedure To add multiple networks to nodes, use the --additional-network argument by running the following command: USD hcp create cluster kubevirt \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --additional-network name:<namespace/name> \ 6 --additional-network name:<namespace/name> 1 Specify the name of your hosted cluster, for instance, example . 2 Specify your worker node count, for example, 2 . 3 Specify the path to your pull secret, for example, /user/name/pullsecret . 4 Specify the memory value, for example, 8Gi . 5 Specify the CPU value, for example, 2 . 6 Set the value of the -additional-network argument to name:<namespace/name> . Replace <namespace/name> with a namespace and name of your NetworkAttachmentDefinitions. 4.3.7.1.1. Using an additional network as default You can add your additional network as a default network for the nodes by disabling the default pod network. Procedure To add an additional network as default to your nodes, run the following command: USD hcp create cluster kubevirt \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --attach-default-network false \ 6 --additional-network name:<namespace>/<network_name> 7 1 Specify the name of your hosted cluster, for instance, example . 2 Specify your worker node count, for example, 2 . 3 Specify the path to your pull secret, for example, /user/name/pullsecret . 4 Specify the memory value, for example, 8Gi . 5 Specify the CPU value, for example, 2 . 6 The --attach-default-network false argument disables the default pod network. 7 Specify the additional network that you want to add to your nodes, for example, name:my-namespace/my-network . 4.3.7.2. Requesting guaranteed CPU resources By default, KubeVirt VMs might share its CPUs with other workloads on a node. This might impact performance of a VM. To avoid the performance impact, you can request a guaranteed CPU access for VMs. Procedure To request guaranteed CPU resources, set the --qos-class argument to Guaranteed by running the following command: USD hcp create cluster kubevirt \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --qos-class Guaranteed 6 1 Specify the name of your hosted cluster, for instance, example . 2 Specify your worker node count, for example, 2 . 3 Specify the path to your pull secret, for example, /user/name/pullsecret . 4 Specify the memory value, for example, 8Gi . 5 Specify the CPU value, for example, 2 . 6 The --qos-class Guaranteed argument guarantees that the specified number of CPU resources are assigned to VMs. 4.3.7.3. Scheduling KubeVirt VMs on a set of nodes By default, KubeVirt VMs created by a node pool are scheduled to any available nodes. You can schedule KubeVirt VMs on a specific set of nodes that has enough capacity to run the VM. Procedure To schedule KubeVirt VMs within a node pool on a specific set of nodes, use the --vm-node-selector argument by running the following command: USD hcp create cluster kubevirt \ --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --vm-node-selector <label_key>=<label_value>,<label_key>=<label_value> 6 1 Specify the name of your hosted cluster, for instance, example . 2 Specify your worker node count, for example, 2 . 3 Specify the path to your pull secret, for example, /user/name/pullsecret . 4 Specify the memory value, for example, 8Gi . 5 Specify the CPU value, for example, 2 . 6 The --vm-node-selector flag defines a specific set of nodes that contains the key-value pairs. Replace <label_key> and <label_value> with the key and value of your labels respectively. 4.3.8. Scaling a node pool You can manually scale a node pool by using the oc scale command. Procedure Run the following command: NODEPOOL_NAME=USD{CLUSTER_NAME}-work NODEPOOL_REPLICAS=5 USD oc scale nodepool/USDNODEPOOL_NAME --namespace clusters \ --replicas=USDNODEPOOL_REPLICAS After a few moments, enter the following command to see the status of the node pool: USD oc --kubeconfig USDCLUSTER_NAME-kubeconfig get nodes Example output NAME STATUS ROLES AGE VERSION example-9jvnf Ready worker 97s v1.27.4+18eadca example-n6prw Ready worker 116m v1.27.4+18eadca example-nc6g4 Ready worker 117m v1.27.4+18eadca example-thp29 Ready worker 4m17s v1.27.4+18eadca example-twxns Ready worker 88s v1.27.4+18eadca 4.3.8.1. Adding node pools You can create node pools for a hosted cluster by specifying a name, number of replicas, and any additional information, such as memory and CPU requirements. Procedure To create a node pool, enter the following information. In this example, the node pool has more CPUs assigned to the VMs: export NODEPOOL_NAME=USD{CLUSTER_NAME}-extra-cpu export WORKER_COUNT="2" export MEM="6Gi" export CPU="4" export DISK="16" USD hcp create nodepool kubevirt \ --cluster-name USDCLUSTER_NAME \ --name USDNODEPOOL_NAME \ --node-count USDWORKER_COUNT \ --memory USDMEM \ --cores USDCPU \ --root-volume-size USDDISK Check the status of the node pool by listing nodepool resources in the clusters namespace: USD oc get nodepools --namespace clusters Example output NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE example example 5 5 False False <4.x.0> example-extra-cpu example 2 False False True True Minimum availability requires 2 replicas, current 0 available Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use. After some time, you can check the status of the node pool by entering the following command: USD oc --kubeconfig USDCLUSTER_NAME-kubeconfig get nodes Example output NAME STATUS ROLES AGE VERSION example-9jvnf Ready worker 97s v1.27.4+18eadca example-n6prw Ready worker 116m v1.27.4+18eadca example-nc6g4 Ready worker 117m v1.27.4+18eadca example-thp29 Ready worker 4m17s v1.27.4+18eadca example-twxns Ready worker 88s v1.27.4+18eadca example-extra-cpu-zh9l5 Ready worker 2m6s v1.27.4+18eadca example-extra-cpu-zr8mj Ready worker 102s v1.27.4+18eadca Verify that the node pool is in the status that you expect by entering this command: USD oc get nodepools --namespace clusters Example output NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE example example 5 5 False False <4.x.0> example-extra-cpu example 2 2 False False <4.x.0> Replace <4.x.0> with the supported OpenShift Container Platform version that you want to use. Additional resources To scale down the data plane to zero, see Scaling down the data plane to zero . 4.3.9. Verifying hosted cluster creation on OpenShift Virtualization To verify that your hosted cluster was successfully created, complete the following steps. Procedure Verify that the HostedCluster resource transitioned to the completed state by entering the following command: USD oc get --namespace clusters hostedclusters <hosted_cluster_name> Example output NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE clusters example 4.12.2 example-admin-kubeconfig Completed True False The hosted control plane is available Verify that all the cluster operators in the hosted cluster are online by entering the following commands: USD hcp create kubeconfig --name <hosted_cluster_name> \ > <hosted_cluster_name>-kubeconfig USD oc get co --kubeconfig=<hosted_cluster_name>-kubeconfig Example output NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.12.2 True False False 2m38s csi-snapshot-controller 4.12.2 True False False 4m3s dns 4.12.2 True False False 2m52s image-registry 4.12.2 True False False 2m8s ingress 4.12.2 True False False 22m kube-apiserver 4.12.2 True False False 23m kube-controller-manager 4.12.2 True False False 23m kube-scheduler 4.12.2 True False False 23m kube-storage-version-migrator 4.12.2 True False False 4m52s monitoring 4.12.2 True False False 69s network 4.12.2 True False False 4m3s node-tuning 4.12.2 True False False 2m22s openshift-apiserver 4.12.2 True False False 23m openshift-controller-manager 4.12.2 True False False 23m openshift-samples 4.12.2 True False False 2m15s operator-lifecycle-manager 4.12.2 True False False 22m operator-lifecycle-manager-catalog 4.12.2 True False False 23m operator-lifecycle-manager-packageserver 4.12.2 True False False 23m service-ca 4.12.2 True False False 4m41s storage 4.12.2 True False False 4m43s 4.4. Deploying hosted control planes on non-bare-metal agent machines You can deploy hosted control planes by configuring a cluster to function as a hosting cluster. The hosting cluster is an OpenShift Container Platform cluster where the control planes are hosted. The hosting cluster is also known as the management cluster. Important Hosted control planes on non-bare-metal agent machines is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Note The management cluster is not the same thing as the managed cluster. A managed cluster is a cluster that the hub cluster manages. The hosted control planes feature is enabled by default. The multicluster engine Operator supports only the default local-cluster managed hub cluster. On Red Hat Advanced Cluster Management (RHACM) 2.10, you can use the local-cluster managed hub cluster as the hosting cluster. A hosted cluster is an OpenShift Container Platform cluster with its API endpoint and control plane that are hosted on the hosting cluster. The hosted cluster includes the control plane and its corresponding data plane. You can use the multicluster engine Operator console or the hcp command-line interface (CLI) to create a hosted cluster. The hosted cluster is automatically imported as a managed cluster. If you want to disable this automatic import feature, see "Disabling the automatic import of hosted clusters into multicluster engine Operator". 4.4.1. Preparing to deploy hosted control planes on non-bare-metal agent machines As you prepare to deploy hosted control planes on bare metal, consider the following information: You can add agent machines as a worker node to a hosted cluster by using the Agent platform. Agent machine represents a host booted with a Discovery Image and ready to be provisioned as an OpenShift Container Platform node. The Agent platform is part of the central infrastructure management service. For more information, see Enabling the central infrastructure management service . All hosts that are not bare metal require a manual boot with a Discovery Image ISO that the central infrastructure management provides. When you scale up the node pool, a machine is created for every replica. For every machine, the Cluster API provider finds and installs an Agent that is approved, is passing validations, is not currently in use, and meets the requirements that are specified in the node pool specification. You can monitor the installation of an Agent by checking its status and conditions. When you scale down a node pool, Agents are unbound from the corresponding cluster. Before you can reuse the Agents, you must restart them by using the Discovery image. When you configure storage for hosted control planes, consider the recommended etcd practices. To ensure that you meet the latency requirements, dedicate a fast storage device to all hosted control planes etcd instances that run on each control-plane node. You can use LVM storage to configure a local storage class for hosted etcd pods. For more information, see "Recommended etcd practices" and "Persistent storage using logical volume manager storage" in the OpenShift Container Platform documentation. 4.4.1.1. Prerequisites for deploying hosted control planes on non-bare-metal agent machines Before you deploy hosted control planes on non-bare-metal agent machines, ensure you meet the following prerequisites: You must have multicluster engine for Kubernetes Operator 2.5 or later installed on an OpenShift Container Platform cluster. You can install the multicluster engine Operator as an Operator from the OpenShift Container Platform OperatorHub. You must have at least one managed OpenShift Container Platform cluster for the multicluster engine Operator. The local-cluster management cluster is automatically imported. For more information about the local-cluster , see Advanced configuration in the Red Hat Advanced Cluster Management documentation. You can check the status of your management cluster by running the following command: USD oc get managedclusters local-cluster You have enabled central infrastructure management. For more information, see Enabling the central infrastructure management service in the Red Hat Advanced Cluster Management documentation. You have installed the hcp command-line interface. Your hosted cluster has a cluster-wide unique name. You are running the management cluster and workers on the same infrastructure. Additional resources Advanced configuration Enabling the central infrastructure management service 4.4.1.2. Firewall, port, and service requirements for non-bare-metal agent machines You must meet the firewall and port requirements so that ports can communicate between the management cluster, the control plane, and hosted clusters. Note Services run on their default ports. However, if you use the NodePort publishing strategy, services run on the port that is assigned by the NodePort service. Use firewall rules, security groups, or other access controls to restrict access to only required sources. Avoid exposing ports publicly unless necessary. For production deployments, use a load balancer to simplify access through a single IP address. A hosted control plane exposes the following services on non-bare-metal agent machines: APIServer The APIServer service runs on port 6443 by default and requires ingress access for communication between the control plane components. If you use MetalLB load balancing, allow ingress access to the IP range that is used for load balancer IP addresses. OAuthServer The OAuthServer service runs on port 443 by default when you use the route and ingress to expose the service. If you use the NodePort publishing strategy, use a firewall rule for the OAuthServer service. Konnectivity The Konnectivity service runs on port 443 by default when you use the route and ingress to expose the service. The Konnectivity agent establishes a reverse tunnel to allow the control plane to access the network for the hosted cluster. The agent uses egress to connect to the Konnectivity server. The server is exposed by using either a route on port 443 or a manually assigned NodePort . If the cluster API server address is an internal IP address, allow access from the workload subnets to the IP address on port 6443. If the address is an external IP address, allow egress on port 6443 to that external IP address from the nodes. Ignition The Ignition service runs on port 443 by default when you use the route and ingress to expose the service. If you use the NodePort publishing strategy, use a firewall rule for the Ignition service. You do not need the following services on non-bare-metal agent machines: OVNSbDb OIDC 4.4.1.3. Infrastructure requirements for non-bare-metal agent machines The Agent platform does not create any infrastructure, but it has the following infrastructure requirements: Agents: An Agent represents a host that is booted with a discovery image and is ready to be provisioned as an OpenShift Container Platform node. DNS: The API and ingress endpoints must be routable. Additional resources Recommended etcd practices Persistent storage using logical volume manager storage Disabling the automatic import of hosted clusters into multicluster engine Operator Manually enabling the hosted control planes feature Disabling the hosted control planes feature Configuring Ansible Automation Platform jobs to run on hosted clusters 4.4.2. Configuring DNS on non-bare-metal agent machines The API Server for the hosted cluster is exposed as a NodePort service. A DNS entry must exist for api.<hosted_cluster_name>.<basedomain> that points to destination where the API Server can be reached. The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane. The entry can also point to a load balancer that is deployed to redirect incoming traffic to the ingress pods. If you are configuring DNS for a connected environment on an IPv4 network, see the following example DNS configuration: api.example.krnl.es. IN A 192.168.122.20 api.example.krnl.es. IN A 192.168.122.21 api.example.krnl.es. IN A 192.168.122.22 api-int.example.krnl.es. IN A 192.168.122.20 api-int.example.krnl.es. IN A 192.168.122.21 api-int.example.krnl.es. IN A 192.168.122.22 `*`.apps.example.krnl.es. IN A 192.168.122.23 If you are configuring DNS for a disconnected environment on an IPv6 network, see the following example DNS configuration: api.example.krnl.es. IN A 2620:52:0:1306::5 api.example.krnl.es. IN A 2620:52:0:1306::6 api.example.krnl.es. IN A 2620:52:0:1306::7 api-int.example.krnl.es. IN A 2620:52:0:1306::5 api-int.example.krnl.es. IN A 2620:52:0:1306::6 api-int.example.krnl.es. IN A 2620:52:0:1306::7 `*`.apps.example.krnl.es. IN A 2620:52:0:1306::10 If you are configuring DNS for a disconnected environment on a dual stack network, be sure to include DNS entries for both IPv4 and IPv6. See the following example DNS configuration: host-record=api-int.hub-dual.dns.base.domain.name,192.168.126.10 host-record=api.hub-dual.dns.base.domain.name,192.168.126.10 address=/apps.hub-dual.dns.base.domain.name/192.168.126.11 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,192.168.126.20 dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,192.168.126.21 dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,192.168.126.22 dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,192.168.126.25 dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,192.168.126.26 host-record=api-int.hub-dual.dns.base.domain.name,2620:52:0:1306::2 host-record=api.hub-dual.dns.base.domain.name,2620:52:0:1306::2 address=/apps.hub-dual.dns.base.domain.name/2620:52:0:1306::3 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5] dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,[2620:52:0:1306::6] dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,[2620:52:0:1306::7] dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,[2620:52:0:1306::8] dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,[2620:52:0:1306::9] 4.4.3. Creating a hosted cluster on non-bare-metal agent machines by using the CLI When you create a hosted cluster with the Agent platform, the HyperShift Operator installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one. As you create a hosted cluster, review the following guidelines: Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it. Do not use clusters as a hosted cluster name. A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster. Procedure Create the hosted control plane namespace by entering the following command: USD oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> 1 1 Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters . Replace <hosted_cluster_name> with your hosted cluster name. Create a hosted cluster by entering the following command: USD hcp create cluster agent \ --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release> \ 10 --node-pool-replicas <node_pool_replica_count> 11 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the path to your pull secret, for example, /user/name/pullsecret . 3 Specify your hosted control plane namespace, for example, clusters-example . Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command. 4 Specify your base domain, for example, krnl.es . 5 The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster. 6 Verify that you have a default storage class configured for your cluster. Otherwise, you might end up with pending PVCs. Specify the etcd storage class name, for example, lvm-storageclass . 7 Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub . 8 Specify your hosted cluster namespace. 9 Specify the availability policy for the hosted control plane components. Supported options are SingleReplica and HighlyAvailable . The default value is HighlyAvailable . 10 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . 11 Specify the node pool replica count, for example, 3 . You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, no node pools are created. Verification After a few moments, verify that your hosted control plane pods are up and running by entering the following command: USD oc -n <hosted_control_plane_namespace> get pods Example output NAME READY STATUS RESTARTS AGE catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s control-plane-operator-f6b4c8465-4k5dh 1/1 Running 0 4m32s Additional resources Manually importing a hosted cluster 4.4.3.1. Creating a hosted cluster on non-bare-metal agent machines by using the web console You can create a hosted cluster on non-bare-metal agent machines by using the OpenShift Container Platform web console. Prerequisites You have access to the cluster with cluster-admin privileges. You have access to the OpenShift Container Platform web console. Procedure Open the OpenShift Container Platform web console and log in by entering your administrator credentials. In the console header, select All Clusters . Click Infrastructure Clusters . Click Create cluster Host inventory Hosted control plane . The Create cluster page is displayed. On the Create cluster page, follow the prompts to enter details about the cluster, node pools, networking, and automation. As you enter details about the cluster, you might find the following tips useful: If you want to use predefined values to automatically populate fields in the console, you can create a host inventory credential. For more information, see Creating a credential for an on-premises environment . On the Cluster details page, the pull secret is your OpenShift Container Platform pull secret that you use to access OpenShift Container Platform resources. If you selected a host inventory credential, the pull secret is automatically populated. On the Node pools page, the namespace contains the hosts for the node pool. If you created a host inventory by using the console, the console creates a dedicated namespace. On the Networking page, you select an API server publishing strategy. The API server for the hosted cluster can be exposed either by using an existing load balancer or as a service of the NodePort type. A DNS entry must exist for the api.<hosted_cluster_name>.<basedomain> setting that points to the destination where the API server can be reached. This entry can be a record that points to one of the nodes in the management cluster or a record that points to a load balancer that redirects incoming traffic to the Ingress pods. Review your entries and click Create . The Hosted cluster view is displayed. Monitor the deployment of the hosted cluster in the Hosted cluster view. If you do not see information about the hosted cluster, ensure that All Clusters is selected, and click the cluster name. Wait until the control plane components are ready. This process can take a few minutes. To view the node pool status, scroll to the NodePool section. The process to install the nodes takes about 10 minutes. You can also click Nodes to confirm whether the nodes joined the hosted cluster. steps To access the web console, see Accessing the web console . 4.4.3.2. Creating a hosted cluster on bare metal by using a mirror registry You can use a mirror registry to create a hosted cluster on bare metal by specifying the --image-content-sources flag in the hcp create cluster command. Procedure Create a YAML file to define Image Content Source Policies (ICSP). See the following example: - mirrors: - brew.registry.redhat.io source: registry.redhat.io - mirrors: - brew.registry.redhat.io source: registry.stage.redhat.io - mirrors: - brew.registry.redhat.io source: registry-proxy.engineering.redhat.com Save the file as icsp.yaml . This file contains your mirror registries. To create a hosted cluster by using your mirror registries, run the following command: USD hcp create cluster agent \ --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --image-content-sources icsp.yaml \ 6 --ssh-key <path_to_ssh_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the path to your pull secret, for example, /user/name/pullsecret . 3 Specify your hosted control plane namespace, for example, clusters-example . Ensure that agents are available in this namespace by using the oc get agent -n <hosted-control-plane-namespace> command. 4 Specify your base domain, for example, krnl.es . 5 The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster. 6 Specify the icsp.yaml file that defines ICSP and your mirror registries. 7 Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub . 8 Specify your hosted cluster namespace. 9 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest . steps To create credentials that you can reuse when you create a hosted cluster with the console, see Creating a credential for an on-premises environment . To access a hosted cluster, see Accessing the hosted cluster . To add hosts to the host inventory by using the Discovery Image, see Adding hosts to the host inventory by using the Discovery Image . To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest . 4.4.4. Verifying hosted cluster creation on non-bare-metal agent machines After the deployment process is complete, you can verify that the hosted cluster was created successfully. Follow these steps a few minutes after you create the hosted cluster. Procedure Obtain the kubeconfig file for your new hosted cluster by entering the following command: USD oc extract -n <hosted_cluster_namespace> \ secret/<hosted_cluster_name>-admin-kubeconfig --to=- \ > kubeconfig-<hosted_cluster_name> Use the kubeconfig file to view the cluster Operators of the hosted cluster. Enter the following command: USD oc get co --kubeconfig=kubeconfig-<hosted_cluster_name> Example output NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.10.26 True False False 2m38s csi-snapshot-controller 4.10.26 True False False 4m3s dns 4.10.26 True False False 2m52s View the running pods on your hosted cluster by entering the following command: USD oc get pods -A --kubeconfig=kubeconfig-<hosted_cluster_name> Example output NAMESPACE NAME READY STATUS RESTARTS AGE kube-system konnectivity-agent-khlqv 0/1 Running 0 3m52s openshift-cluster-samples-operator cluster-samples-operator-6b5bcb9dff-kpnbc 2/2 Running 0 20m openshift-monitoring alertmanager-main-0 6/6 Running 0 100s openshift-monitoring openshift-state-metrics-677b9fb74f-qqp6g 3/3 Running 0 104s 4.5. Deploying hosted control planes on IBM Z You can deploy hosted control planes by configuring a cluster to function as a management cluster. The management cluster is the OpenShift Container Platform cluster where the control planes are hosted. The management cluster is also known as the hosting cluster. Note The management cluster is not the managed cluster. A managed cluster is a cluster that the hub cluster manages. You can convert a managed cluster to a management cluster by using the hypershift add-on to deploy the HyperShift Operator on that cluster. Then, you can start to create the hosted cluster. The multicluster engine Operator supports only the default local-cluster , which is a hub cluster that is managed, and the hub cluster as the management cluster. To provision hosted control planes on bare metal, you can use the Agent platform. The Agent platform uses the central infrastructure management service to add worker nodes to a hosted cluster. For more information, see "Enabling the central infrastructure management service". Each IBM Z system host must be started with the PXE images provided by the central infrastructure management. After each host starts, it runs an Agent process to discover the details of the host and completes the installation. An Agent custom resource represents each host. When you create a hosted cluster with the Agent platform, HyperShift Operator installs the Agent Cluster API provider in the hosted control plane namespace. 4.5.1. Prerequisites to configure hosted control planes on IBM Z The multicluster engine for Kubernetes Operator version 2.5 or later must be installed on an OpenShift Container Platform cluster. You can install multicluster engine Operator as an Operator from the OpenShift Container Platform OperatorHub. The multicluster engine Operator must have at least one managed OpenShift Container Platform cluster. The local-cluster is automatically imported in multicluster engine Operator 2.5 and later. For more information about the local-cluster , see Advanced configuration in the Red Hat Advanced Cluster Management documentation. You can check the status of your hub cluster by running the following command: USD oc get managedclusters local-cluster You need a hosting cluster with at least three worker nodes to run the HyperShift Operator. You need to enable the central infrastructure management service. For more information, see Enabling the central infrastructure management service . You need to install the hosted control plane command line interface. For more information, see Installing the hosted control plane command line interface . Additional resources Advanced configuration Enabling the central infrastructure management service Installing the hosted control planes command-line interface Enabling or disabling the hosted control planes feature 4.5.2. IBM Z infrastructure requirements The Agent platform does not create any infrastructure, but requires the following resources for infrastructure: Agents: An Agent represents a host that is booted with a discovery image, or PXE image and is ready to be provisioned as an OpenShift Container Platform node. DNS: The API and Ingress endpoints must be routable. The hosted control planes feature is enabled by default. If you disabled the feature and want to manually enable it, or if you need to disable the feature, see Enabling or disabling the hosted control planes feature . Additional resources Enabling or disabling the hosted control planes feature 4.5.3. DNS configuration for hosted control planes on IBM Z The API server for the hosted cluster is exposed as a NodePort service. A DNS entry must exist for the api.<hosted_cluster_name>.<base_domain> that points to the destination where the API server is reachable. The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane. The entry can also point to a load balancer deployed to redirect incoming traffic to the Ingress pods. See the following example of a DNS configuration: USD cat /var/named/<example.krnl.es.zone> Example output USD TTL 900 @ IN SOA bastion.example.krnl.es.com. hostmaster.example.krnl.es.com. ( 2019062002 1D 1H 1W 3H ) IN NS bastion.example.krnl.es.com. ; ; api IN A 1xx.2x.2xx.1xx 1 api-int IN A 1xx.2x.2xx.1xx ; ; *.apps IN A 1xx.2x.2xx.1xx ; ;EOF 1 The record refers to the IP address of the API load balancer that handles ingress and egress traffic for hosted control planes. For IBM z/VM, add IP addresses that correspond to the IP address of the agent. compute-0 IN A 1xx.2x.2xx.1yy compute-1 IN A 1xx.2x.2xx.1yy 4.5.4. Creating a hosted cluster on bare metal When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one. As you create a hosted cluster, keep the following guidelines in mind: Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it. Do not use clusters as a hosted cluster name. A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster. The most common service publishing strategy is to expose services through a load balancer. That strategy is the preferred method for exposing the Kubernetes API server. If you create a hosted cluster by using the web console or by using Red Hat Advanced Cluster Management, to set a publishing strategy for a service besides the Kubernetes API server, you must manually specify the servicePublishingStrategy information in the HostedCluster custom resource. Procedure Create the hosted control plane namespace by entering the following command: USD oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters . Replace <hosted_cluster_name> with your hosted cluster name. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command: USD hcp create cluster agent \ --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_public_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 10 --node-pool-replicas <node_pool_replica_count> 11 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the path to your pull secret, for example, /user/name/pullsecret . 3 Specify your hosted control plane namespace, for example, clusters-example . Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command. 4 Specify your base domain, for example, krnl.es . 5 The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster. 6 Specify the etcd storage class name, for example, lvm-storageclass . 7 Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub . 8 Specify your hosted cluster namespace. 9 Specify the availability policy for the hosted control plane components. Supported options are SingleReplica and HighlyAvailable . The default value is HighlyAvailable . 10 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest . 11 Specify the node pool replica count, for example, 3 . You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, no node pools are created. After a few moments, verify that your hosted control plane pods are up and running by entering the following command: USD oc -n <hosted_control_plane_namespace> get pods Example output NAME READY STATUS RESTARTS AGE capi-provider-7dcf5fc4c4-nr9sq 1/1 Running 0 4m32s catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s certified-operators-catalog-884c756c4-zdt64 1/1 Running 0 2m51s cluster-api-f75d86f8c-56wfz 1/1 Running 0 4m32s 4.5.5. Creating an InfraEnv resource for hosted control planes on IBM Z An InfraEnv is an environment where hosts that are booted with PXE images can join as agents. In this case, the agents are created in the same namespace as your hosted control plane. Procedure Create a YAML file to contain the configuration. See the following example: apiVersion: agent-install.openshift.io/v1beta1 kind: InfraEnv metadata: name: <hosted_cluster_name> namespace: <hosted_control_plane_namespace> spec: cpuArchitecture: s390x pullSecretRef: name: pull-secret sshAuthorizedKey: <ssh_public_key> Save the file as infraenv-config.yaml . Apply the configuration by entering the following command: USD oc apply -f infraenv-config.yaml To fetch the URL to download the PXE images, such as, initrd.img , kernel.img , or rootfs.img , which allows IBM Z machines to join as agents, enter the following command: USD oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json 4.5.6. Adding IBM Z agents to the InfraEnv resource To attach compute nodes to a hosted control plane, create agents that help you to scale the node pool. Adding agents in an IBM Z environment requires additional steps, which are described in detail in this section. Unless stated otherwise, these procedures apply to both z/VM and RHEL KVM installations on IBM Z and IBM LinuxONE. 4.5.6.1. Adding IBM Z KVM as agents For IBM Z with KVM, run the following command to start your IBM Z environment with the downloaded PXE images from the InfraEnv resource. After the Agents are created, the host communicates with the Assisted Service and registers in the same namespace as the InfraEnv resource on the management cluster. Procedure Run the following command: virt-install \ --name "<vm_name>" \ 1 --autostart \ --ram=16384 \ --cpu host \ --vcpus=4 \ --location "<path_to_kernel_initrd_image>,kernel=kernel.img,initrd=initrd.img" \ 2 --disk <qcow_image_path> \ 3 --network network:macvtap-net,mac=<mac_address> \ 4 --graphics none \ --noautoconsole \ --wait=-1 --extra-args "rd.neednet=1 nameserver=<nameserver> coreos.live.rootfs_url=http://<http_server>/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs=console=tty1 console=ttyS1,115200n8" 5 1 Specify the name of the virtual machine. 2 Specify the location of the kernel_initrd_image file. 3 Specify the disk image path. 4 Specify the Mac address. 5 Specify the server name of the agents. For ISO boot, download ISO from the InfraEnv resource and boot the nodes by running the following command: virt-install \ --name "<vm_name>" \ 1 --autostart \ --memory=16384 \ --cpu host \ --vcpus=4 \ --network network:macvtap-net,mac=<mac_address> \ 2 --cdrom "<path_to_image.iso>" \ 3 --disk <qcow_image_path> \ --graphics none \ --noautoconsole \ --os-variant <os_version> \ 4 --wait=-1 1 Specify the name of the virtual machine. 2 Specify the Mac address. 3 Specify the location of the image.iso file. 4 Specify the operating system version that you are using. 4.5.6.2. Adding IBM Z LPAR as agents You can add the Logical Partition (LPAR) on IBM Z or IBM LinuxONE as a compute node to a hosted control plane. Procedure Create a boot parameter file for the agents: Example parameter file rd.neednet=1 cio_ignore=all,!condev \ console=ttysclp0 \ ignition.firstboot ignition.platform.id=metal coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \ 1 coreos.inst.persistent-kargs=console=ttysclp0 ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \ 2 rd.znet=qeth,<network_adaptor_range>,layer2=1 rd.<disk_type>=<adapter> \ 3 zfcp.allow_lun_scan=0 ai.ip_cfg_override=1 \ 4 random.trust_cpu=on rd.luks.options=discard 1 For the coreos.live.rootfs_url artifact, specify the matching rootfs artifact for the kernel and initramfs that you are starting. Only HTTP and HTTPS protocols are supported. 2 For the ip parameter, manually assign the IP address, as described in Installing a cluster with z/VM on IBM Z and IBM LinuxONE . 3 For installations on DASD-type disks, use rd.dasd to specify the DASD where Red Hat Enterprise Linux CoreOS (RHCOS) is to be installed. For installations on FCP-type disks, use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed. 4 Specify this parameter when you use an Open Systems Adapter (OSA) or HiperSockets. Download the .ins and initrd.img.addrsize files from the InfraEnv resource. By default, the URL for the .ins and initrd.img.addrsize files is not available in the InfraEnv resource. You must edit the URL to fetch those artifacts. Update the kernel URL endpoint to include ins-file by running the followign command: USD curl -k -L -o generic.ins "< url for ins-file >" Example URL https://.../boot-artifacts/ins-file?arch=s390x&version=4.17.0 Update the initrd URL endpoint to include s390x-initrd-addrsize : Example URL https://..../s390x-initrd-addrsize?api_key=<api-key>&arch=s390x&version=4.17.0 Transfer the initrd , kernel , generic.ins , and initrd.img.addrsize parameter files to the file server. For more information about how to transfer the files with FTP and boot, see "Installing in an LPAR". Start the machine. Repeat the procedure for all other machines in the cluster. Additional resources Installing in an LPAR 4.5.6.3. Adding IBM z/VM as agents If you want to use a static IP for z/VM guest, you must configure the NMStateConfig attribute for the z/VM agent so that the IP parameter persists in the second start. Complete the following steps to start your IBM Z environment with the downloaded PXE images from the InfraEnv resource. After the Agents are created, the host communicates with the Assisted Service and registers in the same namespace as the InfraEnv resource on the management cluster. Procedure Update the parameter file to add the rootfs_url , network_adaptor and disk_type values. Example parameter file rd.neednet=1 cio_ignore=all,!condev \ console=ttysclp0 \ ignition.firstboot ignition.platform.id=metal \ coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \ 1 coreos.inst.persistent-kargs=console=ttysclp0 ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \ 2 rd.znet=qeth,<network_adaptor_range>,layer2=1 rd.<disk_type>=<adapter> \ 3 zfcp.allow_lun_scan=0 ai.ip_cfg_override=1 \ 4 1 For the coreos.live.rootfs_url artifact, specify the matching rootfs artifact for the kernel and initramfs that you are starting. Only HTTP and HTTPS protocols are supported. 2 For the ip parameter, manually assign the IP address, as described in Installing a cluster with z/VM on IBM Z and IBM LinuxONE . 3 For installations on DASD-type disks, use rd.dasd to specify the DASD where Red Hat Enterprise Linux CoreOS (RHCOS) is to be installed. For installations on FCP-type disks, use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHCOS is to be installed. 4 Specify this parameter when you use an Open Systems Adapter (OSA) or HiperSockets. Move initrd , kernel images, and the parameter file to the guest VM by running the following commands: vmur pun -r -u -N kernel.img USDINSTALLERKERNELLOCATION/<image name> vmur pun -r -u -N generic.parm USDPARMFILELOCATION/paramfilename vmur pun -r -u -N initrd.img USDINSTALLERINITRAMFSLOCATION/<image name> Run the following command from the guest VM console: cp ipl c To list the agents and their properties, enter the following command: USD oc -n <hosted_control_plane_namespace> get agents Example output NAME CLUSTER APPROVED ROLE STAGE 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d auto-assign 5e498cd3-542c-e54f-0c58-ed43e28b568a auto-assign Run the following command to approve the agent. USD oc -n <hosted_control_plane_namespace> patch agent \ 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d -p \ '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-zvm-0.hostedn.example.com"}}' \ 1 --type merge 1 Optionally, you can set the agent ID <installation_disk_id> and <hostname> in the specification. Run the following command to verify that the agents are approved: USD oc -n <hosted_control_plane_namespace> get agents Example output NAME CLUSTER APPROVED ROLE STAGE 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d true auto-assign 5e498cd3-542c-e54f-0c58-ed43e28b568a true auto-assign 4.5.7. Scaling the NodePool object for a hosted cluster on IBM Z The NodePool object is created when you create a hosted cluster. By scaling the NodePool object, you can add more compute nodes to the hosted control plane. When you scale up a node pool, a machine is created. The Cluster API provider finds an Agent that is approved, is passing validations, is not currently in use, and meets the requirements that are specified in the node pool specification. You can monitor the installation of an Agent by checking its status and conditions. When you scale down a node pool, Agents are unbound from the corresponding cluster. Before you reuse the clusters, you must boot the clusters by using the PXE image to update the number of nodes. Procedure Run the following command to scale the NodePool object to two nodes: USD oc -n <clusters_namespace> scale nodepool <nodepool_name> --replicas 2 The Cluster API agent provider randomly picks two agents that are then assigned to the hosted cluster. Those agents go through different states and finally join the hosted cluster as OpenShift Container Platform nodes. The agents pass through the transition phases in the following order: binding discovering insufficient installing installing-in-progress added-to-existing-cluster Run the following command to see the status of a specific scaled agent: USD oc -n <hosted_control_plane_namespace> get agent -o \ jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} \ Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}' Example output BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient Run the following command to see the transition phases: USD oc -n <hosted_control_plane_namespace> get agent Example output NAME CLUSTER APPROVED ROLE STAGE 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d hosted-forwarder true auto-assign 5e498cd3-542c-e54f-0c58-ed43e28b568a true auto-assign da503cf1-a347-44f2-875c-4960ddb04091 hosted-forwarder true auto-assign Run the following command to generate the kubeconfig file to access the hosted cluster: USD hcp create kubeconfig \ --namespace <clusters_namespace> \ --name <hosted_cluster_namespace> > <hosted_cluster_name>.kubeconfig After the agents reach the added-to-existing-cluster state, verify that you can see the OpenShift Container Platform nodes by entering the following command: USD oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes Example output NAME STATUS ROLES AGE VERSION worker-zvm-0.hostedn.example.com Ready worker 5m41s v1.24.0+3882f8f worker-zvm-1.hostedn.example.com Ready worker 6m3s v1.24.0+3882f8f Cluster Operators start to reconcile by adding workloads to the nodes. Enter the following command to verify that two machines were created when you scaled up the NodePool object: USD oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io Example output NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION hosted-forwarder-79558597ff-5tbqp hosted-forwarder-crqq5 worker-zvm-0.hostedn.example.com agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d Running 41h 4.15.0 hosted-forwarder-79558597ff-lfjfk hosted-forwarder-crqq5 worker-zvm-1.hostedn.example.com agent://5e498cd3-542c-e54f-0c58-ed43e28b568a Running 41h 4.15.0 Run the following command to check the cluster version: USD oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co Example output NAME VERSION AVAILABLE PROGRESSING SINCE STATUS clusterversion.config.openshift.io/version 4.15.0-ec.2 True False 40h Cluster version is 4.15.0-ec.2 Run the following command to check the cluster operator status: USD oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators For each component of your cluster, the output shows the following cluster operator statuses: NAME , VERSION , AVAILABLE , PROGRESSING , DEGRADED , SINCE , and MESSAGE . For an output example, see Initial Operator configuration . Additional resources Initial Operator configuration 4.6. Deploying hosted control planes on IBM Power You can deploy hosted control planes by configuring a cluster to function as a hosting cluster. The hosting cluster is an OpenShift Container Platform cluster where the control planes are hosted. The hosting cluster is also known as the management cluster. Note The management cluster is not the managed cluster. A managed cluster is a cluster that the hub cluster manages. The multicluster engine Operator supports only the default local-cluster , which is a hub cluster that is managed, and the hub cluster as the hosting cluster. To provision hosted control planes on bare metal, you can use the Agent platform. The Agent platform uses the central infrastructure management service to add worker nodes to a hosted cluster. For more information, see "Enabling the central infrastructure management service". Each IBM Power host must be started with a Discovery Image that the central infrastructure management provides. After each host starts, it runs an Agent process to discover the details of the host and completes the installation. An Agent custom resource represents each host. When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace. 4.6.1. Prerequisites to configure hosted control planes on IBM Power The multicluster engine for Kubernetes Operator version 2.7 and later installed on an OpenShift Container Platform cluster. The multicluster engine Operator is automatically installed when you install Red Hat Advanced Cluster Management (RHACM). You can also install the multicluster engine Operator without RHACM as an Operator from the OpenShift Container Platform OperatorHub. The multicluster engine Operator must have at least one managed OpenShift Container Platform cluster. The local-cluster managed hub cluster is automatically imported in the multicluster engine Operator version 2.7 and later. For more information about local-cluster , see Advanced configuration in the RHACM documentation. You can check the status of your hub cluster by running the following command: USD oc get managedclusters local-cluster You need a hosting cluster with at least 3 worker nodes to run the HyperShift Operator. You need to enable the central infrastructure management service. For more information, see "Enabling the central infrastructure management service". You need to install the hosted control plane command-line interface. For more information, see "Installing the hosted control plane command-line interface". The hosted control planes feature is enabled by default. If you disabled the feature and want to manually enable it, see "Manually enabling the hosted control planes feature". If you need to disable the feature, see "Disabling the hosted control planes feature". Additional resources Advanced configuration Enabling the central infrastructure management service Installing the hosted control planes command-line interface Manually enabling the hosted control planes feature Disabling the hosted control planes feature 4.6.2. IBM Power infrastructure requirements The Agent platform does not create any infrastructure, but requires the following resources for infrastructure: Agents: An Agent represents a host that is booted with a discovery image and is ready to be provisioned as an OpenShift Container Platform node. DNS: The API and Ingress endpoints must be routable. 4.6.3. DNS configuration for hosted control planes on IBM Power The API server for the hosted cluster is exposed. A DNS entry must exist for the api.<hosted_cluster_name>.<basedomain> entry that points to the destination where the API server is reachable. The DNS entry can be as simple as a record that points to one of the nodes in the managed cluster that is running the hosted control plane. The entry can also point to a load balancer that is deployed to redirect incoming traffic to the ingress pods. See the following example of a DNS configuration: USD cat /var/named/<example.krnl.es.zone> Example output USD TTL 900 @ IN SOA bastion.example.krnl.es.com. hostmaster.example.krnl.es.com. ( 2019062002 1D 1H 1W 3H ) IN NS bastion.example.krnl.es.com. ; ; api IN A 1xx.2x.2xx.1xx 1 api-int IN A 1xx.2x.2xx.1xx ; ; *.apps.<hosted-cluster-name>.<basedomain> IN A 1xx.2x.2xx.1xx ; ;EOF 1 The record refers to the IP address of the API load balancer that handles ingress and egress traffic for hosted control planes. For IBM Power, add IP addresses that correspond to the IP address of the agent. Example configuration compute-0 IN A 1xx.2x.2xx.1yy compute-1 IN A 1xx.2x.2xx.1yy 4.6.4. Creating a hosted cluster on bare metal When you create a hosted cluster with the Agent platform, HyperShift installs the Agent Cluster API provider in the hosted control plane namespace. You can create a hosted cluster on bare metal or import one. As you create a hosted cluster, keep the following guidelines in mind: Each hosted cluster must have a cluster-wide unique name. A hosted cluster name cannot be the same as any existing managed cluster in order for multicluster engine Operator to manage it. Do not use clusters as a hosted cluster name. A hosted cluster cannot be created in the namespace of a multicluster engine Operator managed cluster. The most common service publishing strategy is to expose services through a load balancer. That strategy is the preferred method for exposing the Kubernetes API server. If you create a hosted cluster by using the web console or by using Red Hat Advanced Cluster Management, to set a publishing strategy for a service besides the Kubernetes API server, you must manually specify the servicePublishingStrategy information in the HostedCluster custom resource. Procedure Create the hosted control plane namespace by entering the following command: USD oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> Replace <hosted_cluster_namespace> with your hosted cluster namespace name, for example, clusters . Replace <hosted_cluster_name> with your hosted cluster name. Verify that you have a default storage class configured for your cluster. Otherwise, you might see pending PVCs. Run the following command: USD hcp create cluster agent \ --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_public_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 10 --node-pool-replicas <node_pool_replica_count> 11 1 Specify the name of your hosted cluster, for instance, example . 2 Specify the path to your pull secret, for example, /user/name/pullsecret . 3 Specify your hosted control plane namespace, for example, clusters-example . Ensure that agents are available in this namespace by using the oc get agent -n <hosted_control_plane_namespace> command. 4 Specify your base domain, for example, krnl.es . 5 The --api-server-address flag defines the IP address that is used for the Kubernetes API communication in the hosted cluster. If you do not set the --api-server-address flag, you must log in to connect to the management cluster. 6 Specify the etcd storage class name, for example, lvm-storageclass . 7 Specify the path to your SSH public key. The default file path is ~/.ssh/id_rsa.pub . 8 Specify your hosted cluster namespace. 9 Specify the availability policy for the hosted control plane components. Supported options are SingleReplica and HighlyAvailable . The default value is HighlyAvailable . 10 Specify the supported OpenShift Container Platform version that you want to use, for example, 4.18.0-multi . If you are using a disconnected environment, replace <ocp_release_image> with the digest image. To extract the OpenShift Container Platform release image digest, see Extracting the OpenShift Container Platform release image digest . 11 Specify the node pool replica count, for example, 3 . You must specify the replica count as 0 or greater to create the same number of replicas. Otherwise, no node pools are created. After a few moments, verify that your hosted control plane pods are up and running by entering the following command: USD oc -n <hosted_control_plane_namespace> get pods Example output NAME READY STATUS RESTARTS AGE capi-provider-7dcf5fc4c4-nr9sq 1/1 Running 0 4m32s catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s certified-operators-catalog-884c756c4-zdt64 1/1 Running 0 2m51s cluster-api-f75d86f8c-56wfz 1/1 Running 0 4m32s

Commands:
oc get managedclusters local-cluster
aws s3api create-bucket --bucket <bucket_name> \ 1 --create-bucket-configuration LocationConstraint=<region> \ 2 --region <region> 3
aws s3api delete-public-access-block --bucket <bucket_name> 1
echo '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<bucket_name>/*" 1 } ] }' | envsubst > policy.json
aws s3api put-bucket-policy --bucket <bucket_name> \ 1 --policy file://policy.json
oc create secret generic <secret_name> --from-file=credentials=<path>/.aws/credentials --from-literal=bucket=<s3_bucket> --from-literal=region=<region> -n local-cluster
oc label secret hypershift-operator-oidc-provider-s3-credentials -n local-cluster cluster.open-cluster-management.io/backup=true
aws route53 create-hosted-zone --name <basedomain> \ 1 --caller-reference USD(whoami)-USD(date --rfc-3339=date)
aws sts get-caller-identity --query "Arn" --output text
arn:aws:iam::1234567890:user/<aws_username>
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "<arn>" 1 }, "Action": "sts:AssumeRole" } ] }
aws iam create-role --role-name <name> \ 1 --assume-role-policy-document file://<file_name>.json \ 2 --query "Role.Arn"
arn:aws:iam::820196288204:role/myrole
{ "Version": "2012-10-17", "Statement": [ { "Sid": "EC2", "Effect": "Allow", "Action": [ "ec2:CreateDhcpOptions", "ec2:DeleteSubnet", "ec2:ReplaceRouteTableAssociation", "ec2:DescribeAddresses", "ec2:DescribeInstances", "ec2:DeleteVpcEndpoints", "ec2:CreateNatGateway", "ec2:CreateVpc", "ec2:DescribeDhcpOptions", "ec2:AttachInternetGateway", "ec2:DeleteVpcEndpointServiceConfigurations", "ec2:DeleteRouteTable", "ec2:AssociateRouteTable", "ec2:DescribeInternetGateways", "ec2:DescribeAvailabilityZones", "ec2:CreateRoute", "ec2:CreateInternetGateway", "ec2:RevokeSecurityGroupEgress", "ec2:ModifyVpcAttribute", "ec2:DeleteInternetGateway", "ec2:DescribeVpcEndpointConnections", "ec2:RejectVpcEndpointConnections", "ec2:DescribeRouteTables", "ec2:ReleaseAddress", "ec2:AssociateDhcpOptions", "ec2:TerminateInstances", "ec2:CreateTags", "ec2:DeleteRoute", "ec2:CreateRouteTable", "ec2:DetachInternetGateway", "ec2:DescribeVpcEndpointServiceConfigurations", "ec2:DescribeNatGateways", "ec2:DisassociateRouteTable", "ec2:AllocateAddress", "ec2:DescribeSecurityGroups", "ec2:RevokeSecurityGroupIngress", "ec2:CreateVpcEndpoint", "ec2:DescribeVpcs", "ec2:DeleteSecurityGroup", "ec2:DeleteDhcpOptions", "ec2:DeleteNatGateway", "ec2:DescribeVpcEndpoints", "ec2:DeleteVpc", "ec2:CreateSubnet", "ec2:DescribeSubnets" ], "Resource": "*" }, { "Sid": "ELB", "Effect": "Allow", "Action": [ "elasticloadbalancing:DeleteLoadBalancer", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DeleteTargetGroup" ], "Resource": "*" }, { "Sid": "IAMPassRole", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:*:iam::*:role/*-worker-role", "Condition": { "ForAnyValue:StringEqualsIfExists": { "iam:PassedToService": "ec2.amazonaws.com" } } }, { "Sid": "IAM", "Effect": "Allow", "Action": [ "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile", "iam:GetRole", "iam:UpdateAssumeRolePolicy", "iam:GetInstanceProfile", "iam:TagRole", "iam:RemoveRoleFromInstanceProfile", "iam:CreateRole", "iam:DeleteRole", "iam:PutRolePolicy", "iam:AddRoleToInstanceProfile", "iam:CreateOpenIDConnectProvider", "iam:ListOpenIDConnectProviders", "iam:DeleteRolePolicy", "iam:UpdateRole", "iam:DeleteOpenIDConnectProvider", "iam:GetRolePolicy" ], "Resource": "*" }, { "Sid": "Route53", "Effect": "Allow", "Action": [ "route53:ListHostedZonesByVPC", "route53:CreateHostedZone", "route53:ListHostedZones", "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets", "route53:DeleteHostedZone", "route53:AssociateVPCWithHostedZone", "route53:ListHostedZonesByName" ], "Resource": "*" }, { "Sid": "S3", "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets", "s3:ListBucket", "s3:DeleteObject", "s3:DeleteBucket" ], "Resource": "*" } ] }
aws iam put-role-policy --role-name <role_name> \ 1 --policy-name <policy_name> \ 2 --policy-document file://policy.json 3
aws sts get-session-token --output json > sts-creds.json
{ "Credentials": { "AccessKeyId": "ASIA1443CE0GN2ATHWJU", "SecretAccessKey": "XFLN7cZ5AP0d66KhyI4gd8Mu0UCQEDN9cfelW1", "SessionToken": "IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMiJHMEUCIDyipkM7oPKBHiGeI0pMnXst1gDLfs/TvfskXseKCbshAiEAnl1l/Html7Iq9AEIqf////KQburfkq4A3TuppHMr/9j1TgCj1z83SO261bHqlJUazKoy7vBFR/a6LHt55iMBqtKPEsIWjBgj/jSdRJI3j4Gyk1//luKDytcfF/tb9YrxDTPLrACS1lqAxSIFZ82I/jDhbDs=", "Expiration": "2025-05-16T04:19:32+00:00" } }
oc create secret generic <secret_name> --from-literal=aws-access-key-id=<aws_access_key_id> --from-literal=aws-secret-access-key=<aws_secret_access_key> --from-literal=region=<region> -n local-cluster
oc label secret hypershift-operator-private-link-credentials -n local-cluster cluster.open-cluster-management.io/backup=""
oc create secret generic <secret_name> --from-literal=provider=aws --from-literal=domain-filter=<domain_name> --from-file=credentials=<path_to_aws_credentials_file> -n local-cluster
oc label secret hypershift-operator-external-dns-credentials -n local-cluster cluster.open-cluster-management.io/backup=""
dig +short test.user-dest-public.aws.kerberos.com
192.168.1.1
hcp create cluster aws --name=<hosted_cluster_name> --endpoint-access=PublicAndPrivate --external-dns-domain=<public_hosted_zone> ... 1
platform: aws: endpointAccess: PublicAndPrivate services: - service: APIServer servicePublishingStrategy: route: hostname: api-example.service-provider-domain.com type: Route - service: OAuthServer servicePublishingStrategy: route: hostname: oauth-example.service-provider-domain.com type: Route - service: Konnectivity servicePublishingStrategy: type: Route - service: Ignition servicePublishingStrategy: type: Route
export KUBECONFIG=<path_to_management_cluster_kubeconfig>
oc get pod -n hypershift -lapp=external-dns
NAME READY STATUS RESTARTS AGE external-dns-7c89788c69-rn8gp 1/1 Running 0 40s
hcp create cluster aws --role-arn <arn_role> \ 1 --instance-type <instance_type> \ 2 --region <region> \ 3 --auto-repair --generate-ssh --name <hosted_cluster_name> \ 4 --namespace clusters --base-domain <service_consumer_domain> \ 5 --node-pool-replicas <node_replica_count> \ 6 --pull-secret <path_to_your_pull_secret> \ 7 --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 8 --external-dns-domain=<service_provider_domain> \ 9 --endpoint-access=PublicAndPrivate 10 --sts-creds <path_to_sts_credential_file> 11
hcp create cluster aws --name <hosted_cluster_name> \ 1 --infra-id <infra_id> \ 2 --base-domain <basedomain> \ 3 --sts-creds <path_to_sts_credential_file> \ 4 --pull-secret <path_to_pull_secret> \ 5 --region <region> \ 6 --generate-ssh --node-pool-replicas <node_pool_replica_count> \ 7 --namespace <hosted_cluster_namespace> \ 8 --role-arn <role_name> \ 9 --render-into <file_name>.yaml 10
apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool metadata: name: <nodepool_name> 1 spec: platform: aws: placement: tenancy: "default" 2
oc get hostedclusters -n <hosted_cluster_namespace>
oc get nodepools --namespace <hosted_cluster_namespace>
oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes
hcp create kubeconfig --namespace <hosted_cluster_namespace> --name <hosted_cluster_name> > <hosted_cluster_name>.kubeconfig
oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes
hcp create cluster aws --name <hosted_cluster_name> \ 1 --node-pool-replicas=<node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --role-arn <arn_role> \ 5 --region <region> \ 6 --zones <zones> \ 7 --sts-creds <path_to_sts_credential_file> 8
hcp create cluster aws --name <hosted_cluster_name> \ 1 --node-pool-replicas <node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --sts-creds <path_to_sts_credential_file> \ 5 --region <region> \ 6 --role-arn <arn_role> 7
hcp create cluster aws --name <hosted_cluster_name> \ 1 --node-pool-replicas <node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --sts-creds <path_to_sts_credential_file> \ 5 --region <region> \ 6 --release-image quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 7 --role-arn <role_name> 8
hcp create nodepool aws --cluster-name <hosted_cluster_name> \ 1 --name <node_pool_name> \ 2 --node-count <node_pool_replica_count> \ 3 --arch <architecture> 4
hcp create cluster aws --name <hosted_cluster_name> \ 1 --node-pool-replicas=<node_pool_replica_count> \ 2 --base-domain <basedomain> \ 3 --pull-secret <path_to_pull_secret> \ 4 --sts-creds <path_to_sts_credential_file> \ 5 --region <region> \ 6 --endpoint-access Private \ 7 --role-arn <role_name> 8
aws ec2 describe-instances --filter="Name=tag:kubernetes.io/cluster/<infra_id>,Values=owned" | jq '.Reservations[] | .Instances[] | select(.PublicDnsName=="") | .PrivateIpAddress'
hcp create kubeconfig > <hosted_cluster_kubeconfig>
ssh -o ProxyCommand="ssh ec2-user@<bastion_ip> -W %h:%p" core@<node_ip>
mv <path_to_kubeconfig_file> <new_file_name>
export KUBECONFIG=<path_to_kubeconfig_file>
oc get clusteroperators clusterversion
oc get managedclusters local-cluster
api.example.krnl.es. IN A 192.168.122.20 api.example.krnl.es. IN A 192.168.122.21 api.example.krnl.es. IN A 192.168.122.22 api-int.example.krnl.es. IN A 192.168.122.20 api-int.example.krnl.es. IN A 192.168.122.21 api-int.example.krnl.es. IN A 192.168.122.22 `*`.apps.example.krnl.es. IN A 192.168.122.23
api.example.krnl.es. IN A 2620:52:0:1306::5 api.example.krnl.es. IN A 2620:52:0:1306::6 api.example.krnl.es. IN A 2620:52:0:1306::7 api-int.example.krnl.es. IN A 2620:52:0:1306::5 api-int.example.krnl.es. IN A 2620:52:0:1306::6 api-int.example.krnl.es. IN A 2620:52:0:1306::7 `*`.apps.example.krnl.es. IN A 2620:52:0:1306::10
host-record=api-int.hub-dual.dns.base.domain.name,192.168.126.10 host-record=api.hub-dual.dns.base.domain.name,192.168.126.10 address=/apps.hub-dual.dns.base.domain.name/192.168.126.11 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,192.168.126.20 dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,192.168.126.21 dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,192.168.126.22 dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,192.168.126.25 dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,192.168.126.26 host-record=api-int.hub-dual.dns.base.domain.name,2620:52:0:1306::2 host-record=api.hub-dual.dns.base.domain.name,2620:52:0:1306::2 address=/apps.hub-dual.dns.base.domain.name/2620:52:0:1306::3 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5] dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,[2620:52:0:1306::6] dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,[2620:52:0:1306::7] dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,[2620:52:0:1306::8] dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,[2620:52:0:1306::9]
oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>
hcp create cluster agent --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_public_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 10 --node-pool-replicas <node_pool_replica_count> 11
oc -n <hosted_control_plane_namespace> get pods
NAME READY STATUS RESTARTS AGE capi-provider-7dcf5fc4c4-nr9sq 1/1 Running 0 4m32s catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s certified-operators-catalog-884c756c4-zdt64 1/1 Running 0 2m51s cluster-api-f75d86f8c-56wfz 1/1 Running 0 4m32s
- mirrors: - brew.registry.redhat.io source: registry.redhat.io - mirrors: - brew.registry.redhat.io source: registry.stage.redhat.io - mirrors: - brew.registry.redhat.io source: registry-proxy.engineering.redhat.com
hcp create cluster agent --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --image-content-sources icsp.yaml \ 6 --ssh-key <path_to_ssh_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9
oc extract -n <hosted-control-plane-namespace> secret/admin-kubeconfig --to=- > kubeconfig-<hosted-cluster-name>
oc get co --kubeconfig=kubeconfig-<hosted-cluster-name>
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.10.26 True False False 2m38s dns 4.10.26 True False False 2m52s image-registry 4.10.26 True False False 2m8s ingress 4.10.26 True False False 22m
oc get pods -A --kubeconfig=kubeconfig-<hosted-cluster-name>
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system konnectivity-agent-khlqv 0/1 Running 0 3m52s openshift-cluster-node-tuning-operator tuned-dhw5p 1/1 Running 0 109s openshift-cluster-storage-operator cluster-storage-operator-5f784969f5-vwzgz 1/1 Running 1 (113s ago) 20m openshift-cluster-storage-operator csi-snapshot-controller-6b7687b7d9-7nrfw 1/1 Running 0 3m8s openshift-console console-5cbf6c7969-6gk6z 1/1 Running 0 119s openshift-console downloads-7bcd756565-6wj5j 1/1 Running 0 4m3s openshift-dns-operator dns-operator-77d755cd8c-xjfbn 2/2 Running 0 21m openshift-dns dns-default-kfqnh 2/2 Running 0 113s
oc patch ingresscontroller -n openshift-ingress-operator default --type=json -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]'
oc patch storageclass ocs-storagecluster-ceph-rbd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
oc get managedclusters local-cluster
- lastTransitionTime: "2024-10-08T15:38:19Z" message: | 3 of 3 machines are not live migratable Machine user-np-ngst4-gw2hz: DisksNotLiveMigratable: user-np-ngst4-gw2hz is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-gw2hz-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) Machine user-np-ngst4-npq7x: DisksNotLiveMigratable: user-np-ngst4-npq7x is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-npq7x-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) Machine user-np-ngst4-q5nkb: DisksNotLiveMigratable: user-np-ngst4-q5nkb is not a live migratable machine: cannot migrate VMI: PVC user-np-ngst4-q5nkb-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode) observedGeneration: 1 reason: DisksNotLiveMigratable status: "False" type: KubeVirtNodesLiveMigratable
- lastTransitionTime: "2024-10-08T15:38:19Z" message: "All is well" observedGeneration: 1 reason: AsExpected status: "True" type: KubeVirtNodesLiveMigratable
hcp create cluster kubevirt --name <hosted_cluster_name> \ 1 --node-pool-replicas <node_pool_replica_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <value_for_memory> \ 4 --cores <value_for_cpu> \ 5 --etcd-storage-class=<etcd_storage_class> 6
oc -n clusters-<hosted-cluster-name> get pods
NAME READY STATUS RESTARTS AGE capi-provider-5cc7b74f47-n5gkr 1/1 Running 0 3m catalog-operator-5f799567b7-fd6jw 2/2 Running 0 69s certified-operators-catalog-784b9899f9-mrp6p 1/1 Running 0 66s cluster-api-6bbc867966-l4dwl 1/1 Running 0 66s . . . redhat-operators-catalog-9d5fd4d44-z8qqk 1/1 Running 0 66s
oc get --namespace clusters hostedclusters
NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE clusters example <4.x.0> example-admin-kubeconfig Completed True False The hosted control plane is available
hcp create cluster kubevirt --name <hosted-cluster-name> \ 1 --node-pool-replicas <worker-count> \ 2 --pull-secret <path-to-pull-secret> \ 3 --memory <value-for-memory> \ 4 --cores <value-for-cpu> \ 5 --infra-namespace=<hosted-cluster-namespace>-<hosted-cluster-name> \ 6 --infra-kubeconfig-file=<path-to-external-infra-kubeconfig> 7
*.apps.mgmt-cluster.example.com
*.apps.guest.apps.mgmt-cluster.example.com
oc patch ingresscontroller -n openshift-ingress-operator default --type=json -p '[{ "op": "add", "path": "/spec/routeAdmission", "value": {wildcardPolicy: "WildcardsAllowed"}}]'
hcp create cluster kubevirt --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <value_for_memory> \ 4 --cores <value_for_cpu> \ 5 --base-domain <basedomain> 6
oc get --namespace clusters hostedclusters
NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE example example-admin-kubeconfig Partial True False The hosted control plane is available
hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig
oc --kubeconfig <hosted_cluster_name>-kubeconfig get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console <4.x.0> False False False 30m RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.example.hypershift.lab): Get "https://console-openshift-console.apps.example.hypershift.lab": dial tcp: lookup console-openshift-console.apps.example.hypershift.lab on 172.31.0.10:53: no such host ingress <4.x.0> True False True 28m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
oc --kubeconfig <hosted_cluster_name>-kubeconfig get services -n openshift-ingress router-nodeport-default -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}'
oc --kubeconfig <hosted_cluster_name>-kubeconfig get services -n openshift-ingress router-nodeport-default -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}'
apply -f - apiVersion: v1 kind: Service metadata: labels: app: <hosted_cluster_name> name: <hosted_cluster_name>-apps namespace: clusters-<hosted_cluster_name> spec: ports: - name: https-443 port: 443 protocol: TCP targetPort: <https_node_port> 1 - name: http-80 port: 80 protocol: TCP targetPort: <http-node-port> 2 selector: kubevirt.io: virt-launcher type: LoadBalancer
oc -n clusters-<hosted_cluster_name> get service <hosted-cluster-name>-apps -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
192.168.20.30
*.apps.<hosted_cluster_name\>.<base_domain\>.
dig +short test.apps.example.hypershift.lab 192.168.20.30
oc get --namespace clusters hostedclusters
NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE example <4.x.0> example-admin-kubeconfig Completed True False The hosted control plane is available
apiVersion: metallb.io/v1beta1 kind: MetalLB metadata: name: metallb namespace: metallb-system
oc apply -f configure-metallb.yaml
metallb.metallb.io/metallb created
apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: metallb namespace: metallb-system spec: addresses: - 192.168.216.32-192.168.216.122 1
oc apply -f create-ip-address-pool.yaml
ipaddresspool.metallb.io/metallb created
apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: l2advertisement namespace: metallb-system spec: ipAddressPools: - metallb
oc apply -f l2advertisement.yaml
l2advertisement.metallb.io/metallb created
hcp create cluster kubevirt --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --additional-network name:<namespace/name> \ 6 --additional-network name:<namespace/name>
hcp create cluster kubevirt --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --attach-default-network false \ 6 --additional-network name:<namespace>/<network_name> 7
hcp create cluster kubevirt --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --qos-class Guaranteed 6
hcp create cluster kubevirt --name <hosted_cluster_name> \ 1 --node-pool-replicas <worker_node_count> \ 2 --pull-secret <path_to_pull_secret> \ 3 --memory <memory> \ 4 --cores <cpu> \ 5 --vm-node-selector <label_key>=<label_value>,<label_key>=<label_value> 6
NODEPOOL_NAME=USD{CLUSTER_NAME}-work NODEPOOL_REPLICAS=5 oc scale nodepool/USDNODEPOOL_NAME --namespace clusters --replicas=USDNODEPOOL_REPLICAS
oc --kubeconfig USDCLUSTER_NAME-kubeconfig get nodes
NAME STATUS ROLES AGE VERSION example-9jvnf Ready worker 97s v1.27.4+18eadca example-n6prw Ready worker 116m v1.27.4+18eadca example-nc6g4 Ready worker 117m v1.27.4+18eadca example-thp29 Ready worker 4m17s v1.27.4+18eadca example-twxns Ready worker 88s v1.27.4+18eadca
export NODEPOOL_NAME=USD{CLUSTER_NAME}-extra-cpu export WORKER_COUNT="2" export MEM="6Gi" export CPU="4" export DISK="16" hcp create nodepool kubevirt --cluster-name USDCLUSTER_NAME --name USDNODEPOOL_NAME --node-count USDWORKER_COUNT --memory USDMEM --cores USDCPU --root-volume-size USDDISK
oc get nodepools --namespace clusters
NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE example example 5 5 False False <4.x.0> example-extra-cpu example 2 False False True True Minimum availability requires 2 replicas, current 0 available
oc --kubeconfig USDCLUSTER_NAME-kubeconfig get nodes
NAME STATUS ROLES AGE VERSION example-9jvnf Ready worker 97s v1.27.4+18eadca example-n6prw Ready worker 116m v1.27.4+18eadca example-nc6g4 Ready worker 117m v1.27.4+18eadca example-thp29 Ready worker 4m17s v1.27.4+18eadca example-twxns Ready worker 88s v1.27.4+18eadca example-extra-cpu-zh9l5 Ready worker 2m6s v1.27.4+18eadca example-extra-cpu-zr8mj Ready worker 102s v1.27.4+18eadca
oc get nodepools --namespace clusters
NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE example example 5 5 False False <4.x.0> example-extra-cpu example 2 2 False False <4.x.0>
oc get --namespace clusters hostedclusters <hosted_cluster_name>
NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE clusters example 4.12.2 example-admin-kubeconfig Completed True False The hosted control plane is available
hcp create kubeconfig --name <hosted_cluster_name> > <hosted_cluster_name>-kubeconfig
oc get co --kubeconfig=<hosted_cluster_name>-kubeconfig
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.12.2 True False False 2m38s csi-snapshot-controller 4.12.2 True False False 4m3s dns 4.12.2 True False False 2m52s image-registry 4.12.2 True False False 2m8s ingress 4.12.2 True False False 22m kube-apiserver 4.12.2 True False False 23m kube-controller-manager 4.12.2 True False False 23m kube-scheduler 4.12.2 True False False 23m kube-storage-version-migrator 4.12.2 True False False 4m52s monitoring 4.12.2 True False False 69s network 4.12.2 True False False 4m3s node-tuning 4.12.2 True False False 2m22s openshift-apiserver 4.12.2 True False False 23m openshift-controller-manager 4.12.2 True False False 23m openshift-samples 4.12.2 True False False 2m15s operator-lifecycle-manager 4.12.2 True False False 22m operator-lifecycle-manager-catalog 4.12.2 True False False 23m operator-lifecycle-manager-packageserver 4.12.2 True False False 23m service-ca 4.12.2 True False False 4m41s storage 4.12.2 True False False 4m43s
oc get managedclusters local-cluster
api.example.krnl.es. IN A 192.168.122.20 api.example.krnl.es. IN A 192.168.122.21 api.example.krnl.es. IN A 192.168.122.22 api-int.example.krnl.es. IN A 192.168.122.20 api-int.example.krnl.es. IN A 192.168.122.21 api-int.example.krnl.es. IN A 192.168.122.22 `*`.apps.example.krnl.es. IN A 192.168.122.23
api.example.krnl.es. IN A 2620:52:0:1306::5 api.example.krnl.es. IN A 2620:52:0:1306::6 api.example.krnl.es. IN A 2620:52:0:1306::7 api-int.example.krnl.es. IN A 2620:52:0:1306::5 api-int.example.krnl.es. IN A 2620:52:0:1306::6 api-int.example.krnl.es. IN A 2620:52:0:1306::7 `*`.apps.example.krnl.es. IN A 2620:52:0:1306::10
host-record=api-int.hub-dual.dns.base.domain.name,192.168.126.10 host-record=api.hub-dual.dns.base.domain.name,192.168.126.10 address=/apps.hub-dual.dns.base.domain.name/192.168.126.11 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,192.168.126.20 dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,192.168.126.21 dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,192.168.126.22 dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,192.168.126.25 dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,192.168.126.26 host-record=api-int.hub-dual.dns.base.domain.name,2620:52:0:1306::2 host-record=api.hub-dual.dns.base.domain.name,2620:52:0:1306::2 address=/apps.hub-dual.dns.base.domain.name/2620:52:0:1306::3 dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5] dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,[2620:52:0:1306::6] dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,[2620:52:0:1306::7] dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,[2620:52:0:1306::8] dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,[2620:52:0:1306::9]
oc create ns <hosted_cluster_namespace>-<hosted_cluster_name> 1
hcp create cluster agent --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release> \ 10 --node-pool-replicas <node_pool_replica_count> 11
oc -n <hosted_control_plane_namespace> get pods
NAME READY STATUS RESTARTS AGE catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s control-plane-operator-f6b4c8465-4k5dh 1/1 Running 0 4m32s
- mirrors: - brew.registry.redhat.io source: registry.redhat.io - mirrors: - brew.registry.redhat.io source: registry.stage.redhat.io - mirrors: - brew.registry.redhat.io source: registry-proxy.engineering.redhat.com
hcp create cluster agent --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --image-content-sources icsp.yaml \ 6 --ssh-key <path_to_ssh_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> 9
oc extract -n <hosted_cluster_namespace> secret/<hosted_cluster_name>-admin-kubeconfig --to=- > kubeconfig-<hosted_cluster_name>
oc get co --kubeconfig=kubeconfig-<hosted_cluster_name>
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.10.26 True False False 2m38s csi-snapshot-controller 4.10.26 True False False 4m3s dns 4.10.26 True False False 2m52s
oc get pods -A --kubeconfig=kubeconfig-<hosted_cluster_name>
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system konnectivity-agent-khlqv 0/1 Running 0 3m52s openshift-cluster-samples-operator cluster-samples-operator-6b5bcb9dff-kpnbc 2/2 Running 0 20m openshift-monitoring alertmanager-main-0 6/6 Running 0 100s openshift-monitoring openshift-state-metrics-677b9fb74f-qqp6g 3/3 Running 0 104s
oc get managedclusters local-cluster
cat /var/named/<example.krnl.es.zone>
TTL 900 @ IN SOA bastion.example.krnl.es.com. hostmaster.example.krnl.es.com. ( 2019062002 1D 1H 1W 3H ) IN NS bastion.example.krnl.es.com. ; ; api IN A 1xx.2x.2xx.1xx 1 api-int IN A 1xx.2x.2xx.1xx ; ; *.apps IN A 1xx.2x.2xx.1xx ; ;EOF
compute-0 IN A 1xx.2x.2xx.1yy compute-1 IN A 1xx.2x.2xx.1yy
oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>
hcp create cluster agent --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_public_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 10 --node-pool-replicas <node_pool_replica_count> 11
oc -n <hosted_control_plane_namespace> get pods
NAME READY STATUS RESTARTS AGE capi-provider-7dcf5fc4c4-nr9sq 1/1 Running 0 4m32s catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s certified-operators-catalog-884c756c4-zdt64 1/1 Running 0 2m51s cluster-api-f75d86f8c-56wfz 1/1 Running 0 4m32s
apiVersion: agent-install.openshift.io/v1beta1 kind: InfraEnv metadata: name: <hosted_cluster_name> namespace: <hosted_control_plane_namespace> spec: cpuArchitecture: s390x pullSecretRef: name: pull-secret sshAuthorizedKey: <ssh_public_key>
oc apply -f infraenv-config.yaml
oc -n <hosted_control_plane_namespace> get InfraEnv <hosted_cluster_name> -o json
virt-install --name "<vm_name>" \ 1 --autostart --ram=16384 --cpu host --vcpus=4 --location "<path_to_kernel_initrd_image>,kernel=kernel.img,initrd=initrd.img" \ 2 --disk <qcow_image_path> \ 3 --network network:macvtap-net,mac=<mac_address> \ 4 --graphics none --noautoconsole --wait=-1 --extra-args "rd.neednet=1 nameserver=<nameserver> coreos.live.rootfs_url=http://<http_server>/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs=console=tty1 console=ttyS1,115200n8" 5
virt-install --name "<vm_name>" \ 1 --autostart --memory=16384 --cpu host --vcpus=4 --network network:macvtap-net,mac=<mac_address> \ 2 --cdrom "<path_to_image.iso>" \ 3 --disk <qcow_image_path> --graphics none --noautoconsole --os-variant <os_version> \ 4 --wait=-1
rd.neednet=1 cio_ignore=all,!condev console=ttysclp0 ignition.firstboot ignition.platform.id=metal coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \ 1 coreos.inst.persistent-kargs=console=ttysclp0 ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \ 2 rd.znet=qeth,<network_adaptor_range>,layer2=1 rd.<disk_type>=<adapter> \ 3 zfcp.allow_lun_scan=0 ai.ip_cfg_override=1 \ 4 random.trust_cpu=on rd.luks.options=discard
curl -k -L -o generic.ins "< url for ins-file >"
https://.../boot-artifacts/ins-file?arch=s390x&version=4.17.0
https://..../s390x-initrd-addrsize?api_key=<api-key>&arch=s390x&version=4.17.0
rd.neednet=1 cio_ignore=all,!condev console=ttysclp0 ignition.firstboot ignition.platform.id=metal coreos.live.rootfs_url=http://<http_server>/rhcos-<version>-live-rootfs.<architecture>.img \ 1 coreos.inst.persistent-kargs=console=ttysclp0 ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns> \ 2 rd.znet=qeth,<network_adaptor_range>,layer2=1 rd.<disk_type>=<adapter> \ 3 zfcp.allow_lun_scan=0 ai.ip_cfg_override=1 \ 4
vmur pun -r -u -N kernel.img USDINSTALLERKERNELLOCATION/<image name>
vmur pun -r -u -N generic.parm USDPARMFILELOCATION/paramfilename
vmur pun -r -u -N initrd.img USDINSTALLERINITRAMFSLOCATION/<image name>
cp ipl c
oc -n <hosted_control_plane_namespace> get agents
NAME CLUSTER APPROVED ROLE STAGE 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d auto-assign 5e498cd3-542c-e54f-0c58-ed43e28b568a auto-assign
oc -n <hosted_control_plane_namespace> patch agent 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d -p '{"spec":{"installation_disk_id":"/dev/sda","approved":true,"hostname":"worker-zvm-0.hostedn.example.com"}}' \ 1 --type merge
oc -n <hosted_control_plane_namespace> get agents
NAME CLUSTER APPROVED ROLE STAGE 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d true auto-assign 5e498cd3-542c-e54f-0c58-ed43e28b568a true auto-assign
oc -n <clusters_namespace> scale nodepool <nodepool_name> --replicas 2
oc -n <hosted_control_plane_namespace> get agent -o jsonpath='{range .items[*]}BMH: {@.metadata.labels.agent-install\.openshift\.io/bmh} Agent: {@.metadata.name} State: {@.status.debugInfo.state}{"\n"}{end}'
BMH: Agent: 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d State: known-unbound BMH: Agent: 5e498cd3-542c-e54f-0c58-ed43e28b568a State: insufficient
oc -n <hosted_control_plane_namespace> get agent
NAME CLUSTER APPROVED ROLE STAGE 50c23cda-cedc-9bbd-bcf1-9b3a5c75804d hosted-forwarder true auto-assign 5e498cd3-542c-e54f-0c58-ed43e28b568a true auto-assign da503cf1-a347-44f2-875c-4960ddb04091 hosted-forwarder true auto-assign
hcp create kubeconfig --namespace <clusters_namespace> --name <hosted_cluster_namespace> > <hosted_cluster_name>.kubeconfig
oc --kubeconfig <hosted_cluster_name>.kubeconfig get nodes
NAME STATUS ROLES AGE VERSION worker-zvm-0.hostedn.example.com Ready worker 5m41s v1.24.0+3882f8f worker-zvm-1.hostedn.example.com Ready worker 6m3s v1.24.0+3882f8f
oc -n <hosted_control_plane_namespace> get machine.cluster.x-k8s.io
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION hosted-forwarder-79558597ff-5tbqp hosted-forwarder-crqq5 worker-zvm-0.hostedn.example.com agent://50c23cda-cedc-9bbd-bcf1-9b3a5c75804d Running 41h 4.15.0 hosted-forwarder-79558597ff-lfjfk hosted-forwarder-crqq5 worker-zvm-1.hostedn.example.com agent://5e498cd3-542c-e54f-0c58-ed43e28b568a Running 41h 4.15.0
oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusterversion,co
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS clusterversion.config.openshift.io/version 4.15.0-ec.2 True False 40h Cluster version is 4.15.0-ec.2
oc --kubeconfig <hosted_cluster_name>.kubeconfig get clusteroperators
oc get managedclusters local-cluster
cat /var/named/<example.krnl.es.zone>
TTL 900 @ IN SOA bastion.example.krnl.es.com. hostmaster.example.krnl.es.com. ( 2019062002 1D 1H 1W 3H ) IN NS bastion.example.krnl.es.com. ; ; api IN A 1xx.2x.2xx.1xx 1 api-int IN A 1xx.2x.2xx.1xx ; ; *.apps.<hosted-cluster-name>.<basedomain> IN A 1xx.2x.2xx.1xx ; ;EOF
compute-0 IN A 1xx.2x.2xx.1yy compute-1 IN A 1xx.2x.2xx.1yy
oc create ns <hosted_cluster_namespace>-<hosted_cluster_name>
hcp create cluster agent --name=<hosted_cluster_name> \ 1 --pull-secret=<path_to_pull_secret> \ 2 --agent-namespace=<hosted_control_plane_namespace> \ 3 --base-domain=<basedomain> \ 4 --api-server-address=api.<hosted_cluster_name>.<basedomain> \ 5 --etcd-storage-class=<etcd_storage_class> \ 6 --ssh-key <path_to_ssh_public_key> \ 7 --namespace <hosted_cluster_namespace> \ 8 --control-plane-availability-policy HighlyAvailable \ 9 --release-image=quay.io/openshift-release-dev/ocp-release:<ocp_release_image> \ 10 --node-pool-replicas <node_pool_replica_count> 11
oc -n <hosted_control_plane_namespace> get pods
NAME READY STATUS RESTARTS AGE capi-provider-7dcf5fc4c4-nr9sq 1/1 Running 0 4m32s catalog-operator-6cd867cc7-phb2q 2/2 Running 0 2m50s certified-operators-catalog-884c756c4-zdt64 1/1 Running 0 2m51s cluster-api-f75d86f8c-56wfz 1/1 Running 0 4m32s