Chapter 13. Managing machines with the Cluster API 13.1. About the Cluster API Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . The Cluster API is an upstream project that is integrated into OpenShift Container Platform as a Technology Preview for Amazon Web Services (AWS), Google Cloud Platform (GCP), and Red Hat OpenStack Platform (RHOSP). 13.1.1. Cluster API overview You can use the Cluster API to create and manage compute machine sets and compute machines in your OpenShift Container Platform cluster. This capability is in addition or an alternative to managing machines with the Machine API. For OpenShift Container Platform 4.15 clusters, you can use the Cluster API to perform node host provisioning management actions after the cluster installation finishes. This system enables an elastic, dynamic provisioning method on top of public or private cloud infrastructure. With the Cluster API Technology Preview, you can create compute machines and compute machine sets on OpenShift Container Platform clusters for supported providers. You can also explore the features that are enabled by this implementation that might not be available with the Machine API. 13.1.1.1. Cluster API benefits By using the Cluster API, OpenShift Container Platform users and developers gain the following advantages: The option to use upstream community Cluster API infrastructure providers that might not be supported by the Machine API. The opportunity to collaborate with third parties who maintain machine controllers for infrastructure providers. The ability to use the same set of Kubernetes tools for infrastructure management in OpenShift Container Platform. The ability to create compute machine sets by using the Cluster API that support features that are not available with the Machine API. 13.1.1.2. Cluster API limitations Using the Cluster API to manage machines is a Technology Preview feature and has the following limitations: To use this feature, you must enable the TechPreviewNoUpgrade feature set. Important Enabling this feature set cannot be undone and prevents minor version updates. Only AWS, Google Cloud Platform (GCP), and Red Hat OpenStack Platform (RHOSP) clusters can use the Cluster API. You must manually create the primary resources that the Cluster API requires. For more information, see "Getting started with the Cluster API". You cannot use the Cluster API to manage control plane machines. Migration of existing compute machine sets created by the Machine API to Cluster API compute machine sets is not supported. Full feature parity with the Machine API is not available. For clusters that use the Cluster API, OpenShift CLI ( oc ) commands prioritize Cluster API objects over Machine API objects. This behavior impacts any oc command that acts upon any object that is represented in both the Cluster API and the Machine API. For more information and a workaround for this issue, see "Referencing the intended objects when using the CLI" in the troubleshooting content. Additional resources Enabling features using feature gates Getting started with the Cluster API Referencing the intended objects when using the CLI 13.1.2. Cluster API architecture The OpenShift Container Platform integration of the upstream Cluster API is implemented and managed by the Cluster CAPI Operator. The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, in contrast to the Machine API, which uses the openshift-machine-api namespace. 13.1.2.1. The Cluster CAPI Operator The Cluster CAPI Operator is an OpenShift Container Platform Operator that maintains the lifecycle of Cluster API resources. This Operator is responsible for all administrative tasks related to deploying the Cluster API project within an OpenShift Container Platform cluster. If a cluster is configured correctly to allow the use of the Cluster API, the Cluster CAPI Operator installs the Cluster API components on the cluster. For more information, see the "Cluster CAPI Operator" entry in the Cluster Operators reference content. Additional resources Cluster CAPI Operator 13.1.2.2. Cluster API primary resources The Cluster API consists of the following primary resources. For the Technology Preview of this feature, you must create these resources manually in the openshift-cluster-api namespace. Cluster A fundamental unit that represents a cluster that is managed by the Cluster API. Infrastructure cluster A provider-specific resource that defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets. Machine template A provider-specific template that defines the properties of the machines that a compute machine set creates. Machine set A group of machines. Compute machine sets are to machines as replica sets are to pods. To add machines or scale them down, change the replicas field on the compute machine set custom resource to meet your compute needs. With the Cluster API, a compute machine set references a Cluster object and a provider-specific machine template. Machine A fundamental unit that describes the host for a node. The Cluster API creates machines based on the configuration in the machine template. 13.2. Getting started with the Cluster API Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . For the Cluster API Technology Preview, you must create the primary resources that the Cluster API requires manually. 13.2.1. Creating the Cluster API primary resources To create the Cluster API primary resources, you must obtain the cluster ID value, which you use for the <cluster_name> parameter in the cluster resource manifest. 13.2.1.1. Obtaining the cluster ID value You can find the cluster ID value by using the OpenShift CLI ( oc ). Prerequisites You have deployed an OpenShift Container Platform cluster. You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). Procedure Obtain the value of the cluster ID by running the following command: USD oc get infrastructure cluster \ -o jsonpath='{.status.infrastructureName}' You can create the Cluster API primary resources manually by creating YAML manifest files and applying them with the OpenShift CLI ( oc ). 13.2.1.2. Creating the Cluster API cluster resource You can create the cluster resource by creating a YAML manifest file and applying it with the OpenShift CLI ( oc ). Prerequisites You have deployed an OpenShift Container Platform cluster. You have enabled the use of the Cluster API. You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). You have the cluster ID value. Procedure Create a YAML file similar to the following. This procedure uses <cluster_resource_file>.yaml as an example file name. apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: <cluster_name> 1 namespace: openshift-cluster-api spec: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <infrastructure_kind> 2 name: <cluster_name> namespace: openshift-cluster-api 1 Specify the cluster ID as the name of the cluster. 2 Specify the infrastructure kind for the cluster. The following values are valid: AWSCluster : The cluster is running on Amazon Web Services (AWS). GCPCluster : The cluster is running on Google Cloud Platform (GCP). Cluster cloud provider Value Amazon Web Services (AWS) AWSCluster Google Cloud Platform (GCP) GCPCluster Red Hat OpenStack Platform (RHOSP) OpenStackCluster Create the cluster CR by running the following command: USD oc create -f <cluster_resource_file>.yaml Verification Confirm that the cluster CR exists by running the following command: USD oc get cluster Example output NAME PHASE AGE VERSION <cluster_name> Provisioning 4h6m The cluster resource is ready when the value of PHASE is Provisioned . Additional resources Cluster API configuration 13.2.1.3. Creating a Cluster API infrastructure resource You can create a provider-specific infrastructure resource by creating a YAML manifest file and applying it with the OpenShift CLI ( oc ). Prerequisites You have deployed an OpenShift Container Platform cluster. You have enabled the use of the Cluster API. You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). You have the cluster ID value. You have created and applied the cluster resource. Procedure Create a YAML file similar to the following. This procedure uses <infrastructure_resource_file>.yaml as an example file name. apiVersion: infrastructure.cluster.x-k8s.io/<version> 1 kind: <infrastructure_kind> 2 metadata: name: <cluster_name> 3 namespace: openshift-cluster-api spec: 4 1 The apiVersion varies by platform. For more information, see the sample Cluster API infrastructure resource YAML for your provider. The following values are valid: infrastructure.cluster.x-k8s.io/v1beta1 : The version that Amazon Web Services (AWS), Google Cloud Platform (GCP), and Red Hat OpenStack Platform (RHOSP) clusters use. 2 Specify the infrastructure kind for the cluster. This value must match the value for your platform. The following values are valid: Cluster cloud provider Value Amazon Web Services (AWS) AWSCluster Google Cloud Platform (GCP) GCPCluster Red Hat OpenStack Platform (RHOSP) OpenStackCluster 3 Specify the name of the cluster. 4 Specify the details for your environment. These parameters are provider specific. For more information, see the sample Cluster API infrastructure resource YAML for your provider. Create the infrastructure CR by running the following command: USD oc create -f <infrastructure_resource_file>.yaml Verification Confirm that the infrastructure CR is created by running the following command: USD oc get <infrastructure_kind> where <infrastructure_kind> is the value that corresponds to your platform. Example output NAME CLUSTER READY <cluster_name> <cluster_name> true Note This output might contain additional columns that are specific to your cloud provider. Additional resources Sample YAML for a Cluster API infrastructure resource on Amazon Web Services Sample YAML for a Cluster API infrastructure resource on Google Cloud Platform Sample YAML for a Cluster API infrastructure resource on RHOSP 13.2.1.4. Creating a Cluster API machine template You can create a provider-specific machine template resource by creating a YAML manifest file and applying it with the OpenShift CLI ( oc ). Prerequisites You have deployed an OpenShift Container Platform cluster. You have enabled the use of the Cluster API. You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). You have created and applied the cluster and infrastructure resources. Procedure Create a YAML file similar to the following. This procedure uses <machine_template_resource_file>.yaml as an example file name. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <machine_template_kind> 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 1 Specify the machine template kind. This value must match the value for your platform. The following values are valid: Cluster cloud provider Value Amazon Web Services (AWS) AWSMachineTemplate Google Cloud Platform (GCP) GCPMachineTemplate Red Hat OpenStack Platform (RHOSP) OpenStackMachineTemplate 2 Specify a name for the machine template. 3 Specify the details for your environment. These parameters are provider specific. For more information, see the sample Cluster API machine template YAML for your provider. Create the machine template CR by running the following command: USD oc create -f <machine_template_resource_file>.yaml Verification Confirm that the machine template CR is created by running the following command: USD oc get <machine_template_kind> where <machine_template_kind> is the value that corresponds to your platform. Example output NAME AGE <template_name> 77m Additional resources Sample YAML for a Cluster API machine template resource on Amazon Web Services Sample YAML for a Cluster API machine template resource on Google Cloud Platform Sample YAML for a Cluster API machine template resource on RHOSP 13.2.1.5. Creating a Cluster API compute machine set You can create compute machine sets that use the Cluster API to dynamically manage the machine compute resources for specific workloads of your choice. Prerequisites You have deployed an OpenShift Container Platform cluster. You have enabled the use of the Cluster API. You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). You have created the cluster, infrastructure, and machine template resources. Procedure Create a YAML file similar to the following. This procedure uses <machine_set_resource_file>.yaml as an example file name. apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: 3 # ... 1 Specify a name for the compute machine set. 2 Specify the name of the cluster. 3 Specify the details for your environment. These parameters are provider specific. For more information, see the sample Cluster API compute machine set YAML for your provider. Create the compute machine set CR by running the following command: USD oc create -f <machine_set_resource_file>.yaml Confirm that the compute machine set CR is created by running the following command: USD oc get machineset -n openshift-cluster-api 1 1 Specify the openshift-cluster-api namespace. Example output NAME CLUSTER REPLICAS READY AVAILABLE AGE VERSION <machine_set_name> <cluster_name> 1 1 1 17m When the new compute machine set is available, the REPLICAS and AVAILABLE values match. If the compute machine set is not available, wait a few minutes and run the command again. Verification To verify that the compute machine set is creating machines according to your required configuration, review the lists of machines and nodes in the cluster by running the following commands: View the list of Cluster API machines: USD oc get machine -n openshift-cluster-api 1 1 Specify the openshift-cluster-api namespace. Example output NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_set_name>-<string_id> <cluster_name> <ip_address>.<region>.compute.internal <provider_id> Running 8m23s View the list of nodes: USD oc get node Example output NAME STATUS ROLES AGE VERSION <ip_address_1>.<region>.compute.internal Ready worker 5h14m v1.28.5 <ip_address_2>.<region>.compute.internal Ready master 5h19m v1.28.5 <ip_address_3>.<region>.compute.internal Ready worker 7m v1.28.5 Additional resources Sample YAML for a Cluster API compute machine set resource on Amazon Web Services Sample YAML for a Cluster API compute machine set resource on Google Cloud Platform Sample YAML for a Cluster API compute machine set resource on RHOSP 13.3. Managing machines with the Cluster API Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . 13.3.1. Modifying a Cluster API machine template You can update the machine template resource for your cluster by modifying the YAML manifest file and applying it with the OpenShift CLI ( oc ). Prerequisites You have deployed an OpenShift Container Platform cluster that uses the Cluster API. You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). Procedure List the machine template resource for your cluster by running the following command: USD oc get <machine_template_kind> 1 1 Specify the value that corresponds to your platform. The following values are valid: Cluster cloud provider Value Amazon Web Services (AWS) AWSMachineTemplate Google Cloud Platform (GCP) GCPMachineTemplate Red Hat OpenStack Platform (RHOSP) OpenStackMachineTemplate Example output NAME AGE <template_name> 77m Write the machine template resource for your cluster to a file that you can edit by running the following command: USD oc get <machine_template_kind> <template_name> -o yaml > <template_name>.yaml where <template_name> is the name of the machine template resource for your cluster. Make a copy of the <template_name>.yaml file with a different name. This procedure uses <modified_template_name>.yaml as an example file name. Use a text editor to make changes to the <modified_template_name>.yaml file that defines the updated machine template resource for your cluster. When editing the machine template resource, observe the following: The parameters in the spec stanza are provider specific. For more information, see the sample Cluster API machine template YAML for your provider. You must use a value for the metadata.name parameter that differs from any existing values. Important For any Cluster API compute machine sets that reference this template, you must update the spec.template.spec.infrastructureRef.name parameter to match the metadata.name value in the new machine template resource. Apply the machine template CR by running the following command: USD oc apply -f <modified_template_name>.yaml 1 1 Use the edited YAML file with a new name. steps For any Cluster API compute machine sets that reference this template, update the spec.template.spec.infrastructureRef.name parameter to match the metadata.name value in the new machine template resource. For more information, see "Modifying a compute machine set by using the CLI." Additional resources Sample YAML for a Cluster API machine template resource on Amazon Web Services Sample YAML for a Cluster API machine template resource on Google Cloud Platform Sample YAML for a Cluster API machine template resource on RHOSP Modifying a compute machine set by using the CLI 13.3.2. Modifying a compute machine set by using the CLI You can modify the configuration of a compute machine set, and then propagate the changes to the machines in your cluster by using the CLI. By updating the compute machine set configuration, you can enable features or change the properties of the machines it creates. When you modify a compute machine set, your changes only apply to compute machines that are created after you save the updated MachineSet custom resource (CR). The changes do not affect existing machines. Note Changes made in the underlying cloud provider are not reflected in the Machine or MachineSet CRs. To adjust instance configuration in cluster-managed infrastructure, use the cluster-side resources. You can replace the existing machines with new ones that reflect the updated configuration by scaling the compute machine set to create twice the number of replicas and then scaling it down to the original number of replicas. If you need to scale a compute machine set without making other changes, you do not need to delete the machines. Note By default, the OpenShift Container Platform router pods are deployed on compute machines. Because the router is required to access some cluster resources, including the web console, do not scale the compute machine set to 0 unless you first relocate the router pods. The output examples in this procedure use the values for an AWS cluster. Prerequisites Your OpenShift Container Platform cluster uses the Cluster API. You are logged in to the cluster as an administrator by using the OpenShift CLI ( oc ). Procedure List the compute machine sets in your cluster by running the following command: USD oc get machinesets.cluster.x-k8s.io -n openshift-cluster-api Example output NAME CLUSTER REPLICAS READY AVAILABLE AGE VERSION <compute_machine_set_name_1> <cluster_name> 1 1 1 26m <compute_machine_set_name_2> <cluster_name> 1 1 1 26m Edit a compute machine set by running the following command: USD oc edit machinesets.cluster.x-k8s.io <machine_set_name> \ -n openshift-cluster-api Note the value of the spec.replicas field, because you need it when scaling the machine set to apply the changes. apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> namespace: openshift-cluster-api spec: replicas: 2 1 # ... 1 The examples in this procedure show a compute machine set that has a replicas value of 2 . Update the compute machine set CR with the configuration options that you want and save your changes. List the machines that are managed by the updated compute machine set by running the following command: USD oc get machines.cluster.x-k8s.io \ -n openshift-cluster-api \ -l cluster.x-k8s.io/set-name=<machine_set_name> Example output for an AWS cluster NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_original_1> <cluster_name> <original_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h <machine_name_original_2> <cluster_name> <original_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h For each machine that is managed by the updated compute machine set, set the delete annotation by running the following command: USD oc annotate machines.cluster.x-k8s.io/<machine_name_original_1> \ -n openshift-cluster-api \ cluster.x-k8s.io/delete-machine="true" To create replacement machines with the new configuration, scale the compute machine set to twice the number of replicas by running the following command: USD oc scale --replicas=4 \ 1 machinesets.cluster.x-k8s.io <machine_set_name> \ -n openshift-cluster-api 1 The original example value of 2 is doubled to 4 . List the machines that are managed by the updated compute machine set by running the following command: USD oc get machines.cluster.x-k8s.io \ -n openshift-cluster-api \ -l cluster.x-k8s.io/set-name=<machine_set_name> Example output for an AWS cluster NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_original_1> <cluster_name> <original_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h <machine_name_original_2> <cluster_name> <original_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h <machine_name_updated_1> <cluster_name> <updated_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Provisioned 55s <machine_name_updated_2> <cluster_name> <updated_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Provisioning 55s When the new machines are in the Running phase, you can scale the compute machine set to the original number of replicas. To remove the machines that were created with the old configuration, scale the compute machine set to the original number of replicas by running the following command: USD oc scale --replicas=2 \ 1 machinesets.cluster.x-k8s.io <machine_set_name> \ -n openshift-cluster-api 1 The original example value of 2 . Verification To verify that a machine created by the updated machine set has the correct configuration, examine the relevant fields in the CR for one of the new machines by running the following command: USD oc describe machines.cluster.x-k8s.io <machine_name_updated_1> \ -n openshift-cluster-api To verify that the compute machines without the updated configuration are deleted, list the machines that are managed by the updated compute machine set by running the following command: USD oc get machines.cluster.x-k8s.io \ -n openshift-cluster-api \ cluster.x-k8s.io/set-name=<machine_set_name> Example output while deletion is in progress for an AWS cluster NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_original_1> <cluster_name> <original_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_original_2> <cluster_name> <original_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_updated_1> <cluster_name> <updated_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_updated_2> <cluster_name> <updated_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m Example output when deletion is complete for an AWS cluster NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_updated_1> <cluster_name> <updated_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_updated_2> <cluster_name> <updated_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m Additional resources Sample YAML for a Cluster API compute machine set resource on Amazon Web Services Sample YAML for a Cluster API compute machine set resource on Google Cloud Platform Sample YAML for a Cluster API compute machine set resource on RHOSP 13.4. Cluster API configuration Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . The following example YAML files show how to make the Cluster API primary resources work together and configure settings for the machines that they create that are appropriate for your environment. 13.4.1. Sample YAML for a Cluster API cluster resource The cluster resource defines the name and infrastructure provider for the cluster and is managed by the Cluster API. This resource has the same structure for all providers. apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: <cluster_name> 1 namespace: openshift-cluster-api spec: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <infrastructure_kind> 2 name: <cluster_name> namespace: openshift-cluster-api 1 Specify the name of the cluster. 2 Specify the infrastructure kind for the cluster. The following values are valid: Cluster cloud provider Value Amazon Web Services AWSCluster GCP GCPCluster RHOSP OpenStackCluster 13.4.2. Provider-specific configuration options The remaining Cluster API resources are provider-specific. For provider-specific configuration options for your cluster, see the following resources: Cluster API configuration options for Amazon Web Services Cluster API configuration options for Google Cloud Platform Cluster API configuration options for RHOSP 13.5. Configuration options for Cluster API machines 13.5.1. Cluster API configuration options for Amazon Web Services Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . You can change the configuration of your Amazon Web Services (AWS) Cluster API machines by updating values in the Cluster API custom resource manifests. 13.5.1.1. Sample YAML for configuring Amazon Web Services clusters The following example YAML files show configurations for an Amazon Web Services cluster. 13.5.1.1.1. Sample YAML for a Cluster API infrastructure resource on Amazon Web Services The infrastructure resource is provider-specific and defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets. The compute machine set references this resource when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSCluster 1 metadata: name: <cluster_name> 2 namespace: openshift-cluster-api spec: controlPlaneEndpoint: 3 host: <control_plane_endpoint_address> port: 6443 region: <region> 4 1 Specify the infrastructure kind for the cluster. This value must match the value for your platform. 2 Specify the cluster ID as the name of the cluster. 3 Specify the address of the control plane endpoint and the port to use to access it. 4 Specify the AWS region. 13.5.1.1.2. Sample YAML for a Cluster API machine template resource on Amazon Web Services The machine template resource is provider-specific and defines the basic properties of the machines that a compute machine set creates. The compute machine set references this template when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 uncompressedUserData: true iamInstanceProfile: # ... instanceType: m5.large cloudInit: insecureSkipSecretsManager: true ami: id: # ... subnet: filters: - name: tag:Name values: - # ... additionalSecurityGroups: - filters: - name: tag:Name values: - # ... 1 Specify the machine template kind. This value must match the value for your platform. 2 Specify a name for the machine template. 3 Specify the details for your environment. The values here are examples. 13.5.1.1.3. Sample YAML for a Cluster API compute machine set resource on Amazon Web Services The compute machine set resource defines additional properties of the machines that it creates. The compute machine set also references the infrastructure resource and machine template when creating machines. apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSMachineTemplate 4 name: <template_name> 5 1 Specify a name for the compute machine set. 2 Specify the cluster ID as the name of the cluster. 3 For the Cluster API Technology Preview, the Operator can use the worker user data secret from the openshift-machine-api namespace. 4 Specify the machine template kind. This value must match the value for your platform. 5 Specify the machine template name. 13.5.2. Cluster API configuration options for Google Cloud Platform Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . You can change the configuration of your Google Cloud Platform (GCP) Cluster API machines by updating values in the Cluster API custom resource manifests. 13.5.2.1. Sample YAML for configuring Google Cloud Platform clusters The following example YAML files show configurations for a Google Cloud Platform cluster. 13.5.2.1.1. Sample YAML for a Cluster API infrastructure resource on Google Cloud Platform The infrastructure resource is provider-specific and defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets. The compute machine set references this resource when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPCluster 1 metadata: name: <cluster_name> 2 spec: controlPlaneEndpoint: 3 host: <control_plane_endpoint_address> port: 6443 network: name: <cluster_name>-network project: <project> 4 region: <region> 5 1 Specify the infrastructure kind for the cluster. This value must match the value for your platform. 2 Specify the cluster ID as the name of the cluster. 3 Specify the IP address of the control plane endpoint and the port used to access it. 4 Specify the GCP project name. 5 Specify the GCP region. 13.5.2.1.2. Sample YAML for a Cluster API machine template resource on Google Cloud Platform The machine template resource is provider-specific and defines the basic properties of the machines that a compute machine set creates. The compute machine set references this template when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 rootDeviceType: pd-ssd rootDeviceSize: 128 instanceType: n1-standard-4 image: projects/rhcos-cloud/global/images/rhcos-411-85-202203181601-0-gcp-x86-64 subnet: <cluster_name>-worker-subnet serviceAccounts: email: <service_account_email_address> scopes: - https://www.googleapis.com/auth/cloud-platform additionalLabels: kubernetes-io-cluster-<cluster_name>: owned additionalNetworkTags: - <cluster_name>-worker ipForwarding: Disabled 1 Specify the machine template kind. This value must match the value for your platform. 2 Specify a name for the machine template. 3 Specify the details for your environment. The values here are examples. 13.5.2.1.3. Sample YAML for a Cluster API compute machine set resource on Google Cloud Platform The compute machine set resource defines additional properties of the machines that it creates. The compute machine set also references the infrastructure resource and machine template when creating machines. apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 4 name: <template_name> 5 failureDomain: <failure_domain> 6 1 Specify a name for the compute machine set. 2 Specify the cluster ID as the name of the cluster. 3 For the Cluster API Technology Preview, the Operator can use the worker user data secret from the openshift-machine-api namespace. 4 Specify the machine template kind. This value must match the value for your platform. 5 Specify the machine template name. 6 Specify the failure domain within the GCP region. 13.5.3. Cluster API configuration options for Red Hat OpenStack Platform Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . You can change the configuration of your Red Hat OpenStack Platform (RHOSP) Cluster API machines by updating values in the Cluster API custom resource manifests. 13.5.3.1. Sample YAML for configuring RHOSP clusters The following example YAML files show configurations for a RHOSP cluster. 13.5.3.1.1. Sample YAML for a Cluster API infrastructure cluster resource on RHOSP The infrastructure cluster resource is provider-specific and defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets. The compute machine set references this resource when creating machines. Important Only the parameters in the following example are validated to be compatible with OpenShift Container Platform. Other parameters, such as those documented in the upstream Cluster API Provider OpenStack book, might cause undesired behavior. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: OpenStackCluster 1 metadata: name: <cluster_name> 2 namespace: openshift-cluster-api labels: cluster.x-k8s.io/cluster-name: <cluster_name> spec: controlPlaneEndpoint: <control_plane_endpoint_address> 3 disableAPIServerFloatingIP: true tags: - openshiftClusterID=<cluster_name> network: id: <api_service_network_id> 4 externalNetwork: id: <floating_network_id> 5 identityRef: cloudName: openstack name: openstack-cloud-credentials 1 Specify the infrastructure kind for the cluster. This value must match the value for your platform. 2 Specify the cluster ID as the name of the cluster. 3 Specify the IP address of the control plane endpoint and the port used to access it. 4 Specify the UUID of the default network to use for machines that do not specify ports. Note This feature might be removed in a future release. To prevent issues due to the removal of this feature, specify at least one port in the machine specification instead of relying solely on this feature. 5 Specify the UUID of an external network. You can specify the network used to assign external floating IP addresses or the network used for egress for this field. Note The infrastructure cluster resource requires this field but does not currently use this value. This requirement is planned to be removed in a future release. 13.5.3.1.2. Sample YAML for a Cluster API machine template resource on RHOSP The machine template resource is provider-specific and defines the basic properties of the machines that a compute machine set creates. The compute machine set references this template when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: OpenStackMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 flavor: <openstack_node_machine_flavor> 4 image: filter: name: <openstack_image> 5 1 Specify the machine template kind. This value must match the value for your platform. 2 Specify a name for the machine template. 3 Specify the details for your environment. The values here are examples. 4 Specify the RHOSP flavor to use. For more information, see Creating flavors for launching instances . 5 Specify the image to use. 13.5.3.1.3. Sample YAML for a Cluster API compute machine set resource on RHOSP The compute machine set resource defines additional properties of the machines that it creates. The compute machine set also references the infrastructure resource and machine template when creating machines. apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example cluster.x-k8s.io/cluster-name: <cluster_name> cluster.x-k8s.io/set-name: <machine_set_name> template: metadata: labels: test: example cluster.x-k8s.io/cluster-name: <cluster_name> cluster.x-k8s.io/set-name: <machine_set_name> node-role.kubernetes.io/<role>: "" spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: OpenStackMachineTemplate 4 name: <template_name> 5 failureDomain: <nova_availability_zone> 6 1 Specify a name for the compute machine set. 2 Specify the cluster ID as the name of the cluster. 3 For the Cluster API Technology Preview, the Operator can use the worker user data secret from the openshift-machine-api namespace. 4 Specify the machine template kind. This value must match the value for your platform. 5 Specify the machine template name. 6 Optional: Specify the name of the Nova availability zone for the machine set to create machines in. If you do not specify a value, machines are not restricted to a specific availability zone. 13.6. Troubleshooting clusters that use the Cluster API Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Use the information in this section to understand and recover from issues you might encounter. Generally, troubleshooting steps for problems with the Cluster API are similar to those steps for problems with the Machine API. The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, whereas the Machine API uses the openshift-machine-api namespace. When using oc commands that reference a namespace, be sure to reference the correct one. 13.6.1. Referencing the intended objects when using the CLI For clusters that use the Cluster API, OpenShift CLI ( oc ) commands prioritize Cluster API objects over Machine API objects. This behavior impacts any oc command that acts upon any object that is represented in both the Cluster API and the Machine API. This explanation uses the oc delete machine command, which deletes a machine, as an example. Cause When you run an oc command, oc communicates with the Kube API server to determine which objects to act upon. The Kube API server uses the first installed custom resource definition (CRD) it encounters alphabetically when an oc command is run. CRDs for Cluster API objects are in the cluster.x-k8s.io group, while CRDs for Machine API objects are in the machine.openshift.io group. Because the letter c precedes the letter m alphabetically, the Kube API server matches on the Cluster API object CRD. As a result, the oc command acts upon Cluster API objects. Consequences Due to this behavior, the following unintended outcomes can occur on a cluster that uses the Cluster API: For namespaces that contain both types of objects, commands such as oc get machine return only Cluster API objects. For namespaces that contain only Machine API objects, commands such as oc get machine return no results. Workaround You can ensure that oc commands act on the type of objects you intend by using the corresponding fully qualified name. Prerequisites You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). Procedure To delete a Machine API machine, use the fully qualified name machine.machine.openshift.io when running the oc delete machine command: USD oc delete machine.machine.openshift.io <machine_name> To delete a Cluster API machine, use the fully qualified name machine.cluster.x-k8s.io when running the oc delete machine command: USD oc delete machine.cluster.x-k8s.io <machine_name>

Commands:
oc get infrastructure cluster -o jsonpath='{.status.infrastructureName}'
apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: <cluster_name> 1 namespace: openshift-cluster-api spec: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <infrastructure_kind> 2 name: <cluster_name> namespace: openshift-cluster-api
oc create -f <cluster_resource_file>.yaml
oc get cluster
NAME PHASE AGE VERSION <cluster_name> Provisioning 4h6m
apiVersion: infrastructure.cluster.x-k8s.io/<version> 1 kind: <infrastructure_kind> 2 metadata: name: <cluster_name> 3 namespace: openshift-cluster-api spec: 4
oc create -f <infrastructure_resource_file>.yaml
oc get <infrastructure_kind>
NAME CLUSTER READY <cluster_name> <cluster_name> true
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <machine_template_kind> 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3
oc create -f <machine_template_resource_file>.yaml
oc get <machine_template_kind>
NAME AGE <template_name> 77m
apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: 3
oc create -f <machine_set_resource_file>.yaml
oc get machineset -n openshift-cluster-api 1
NAME CLUSTER REPLICAS READY AVAILABLE AGE VERSION <machine_set_name> <cluster_name> 1 1 1 17m
oc get machine -n openshift-cluster-api 1
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_set_name>-<string_id> <cluster_name> <ip_address>.<region>.compute.internal <provider_id> Running 8m23s
oc get node
NAME STATUS ROLES AGE VERSION <ip_address_1>.<region>.compute.internal Ready worker 5h14m v1.28.5 <ip_address_2>.<region>.compute.internal Ready master 5h19m v1.28.5 <ip_address_3>.<region>.compute.internal Ready worker 7m v1.28.5
oc get <machine_template_kind> 1
NAME AGE <template_name> 77m
oc get <machine_template_kind> <template_name> -o yaml > <template_name>.yaml
oc apply -f <modified_template_name>.yaml 1
oc get machinesets.cluster.x-k8s.io -n openshift-cluster-api
NAME CLUSTER REPLICAS READY AVAILABLE AGE VERSION <compute_machine_set_name_1> <cluster_name> 1 1 1 26m <compute_machine_set_name_2> <cluster_name> 1 1 1 26m
oc edit machinesets.cluster.x-k8s.io <machine_set_name> -n openshift-cluster-api
apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> namespace: openshift-cluster-api spec: replicas: 2 1
oc get machines.cluster.x-k8s.io -n openshift-cluster-api -l cluster.x-k8s.io/set-name=<machine_set_name>
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_original_1> <cluster_name> <original_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h <machine_name_original_2> <cluster_name> <original_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h
oc annotate machines.cluster.x-k8s.io/<machine_name_original_1> -n openshift-cluster-api cluster.x-k8s.io/delete-machine="true"
oc scale --replicas=4 \ 1 machinesets.cluster.x-k8s.io <machine_set_name> -n openshift-cluster-api
oc get machines.cluster.x-k8s.io -n openshift-cluster-api -l cluster.x-k8s.io/set-name=<machine_set_name>
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_original_1> <cluster_name> <original_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h <machine_name_original_2> <cluster_name> <original_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 4h <machine_name_updated_1> <cluster_name> <updated_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Provisioned 55s <machine_name_updated_2> <cluster_name> <updated_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Provisioning 55s
oc scale --replicas=2 \ 1 machinesets.cluster.x-k8s.io <machine_set_name> -n openshift-cluster-api
oc describe machines.cluster.x-k8s.io <machine_name_updated_1> -n openshift-cluster-api
oc get machines.cluster.x-k8s.io -n openshift-cluster-api cluster.x-k8s.io/set-name=<machine_set_name>
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_original_1> <cluster_name> <original_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_original_2> <cluster_name> <original_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_updated_1> <cluster_name> <updated_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_updated_2> <cluster_name> <updated_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_name_updated_1> <cluster_name> <updated_1_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m <machine_name_updated_2> <cluster_name> <updated_2_ip>.<region>.compute.internal aws:///us-east-2a/i-04e7b2cbd61fd2075 Running 18m
apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: <cluster_name> 1 namespace: openshift-cluster-api spec: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <infrastructure_kind> 2 name: <cluster_name> namespace: openshift-cluster-api
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSCluster 1 metadata: name: <cluster_name> 2 namespace: openshift-cluster-api spec: controlPlaneEndpoint: 3 host: <control_plane_endpoint_address> port: 6443 region: <region> 4
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 uncompressedUserData: true iamInstanceProfile: # instanceType: m5.large cloudInit: insecureSkipSecretsManager: true ami: id: # subnet: filters: - name: tag:Name values: - # additionalSecurityGroups: - filters: - name: tag:Name values: - #
apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSMachineTemplate 4 name: <template_name> 5
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPCluster 1 metadata: name: <cluster_name> 2 spec: controlPlaneEndpoint: 3 host: <control_plane_endpoint_address> port: 6443 network: name: <cluster_name>-network project: <project> 4 region: <region> 5
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 rootDeviceType: pd-ssd rootDeviceSize: 128 instanceType: n1-standard-4 image: projects/rhcos-cloud/global/images/rhcos-411-85-202203181601-0-gcp-x86-64 subnet: <cluster_name>-worker-subnet serviceAccounts: email: <service_account_email_address> scopes: - https://www.googleapis.com/auth/cloud-platform additionalLabels: kubernetes-io-cluster-<cluster_name>: owned additionalNetworkTags: - <cluster_name>-worker ipForwarding: Disabled
apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 4 name: <template_name> 5 failureDomain: <failure_domain> 6
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: OpenStackCluster 1 metadata: name: <cluster_name> 2 namespace: openshift-cluster-api labels: cluster.x-k8s.io/cluster-name: <cluster_name> spec: controlPlaneEndpoint: <control_plane_endpoint_address> 3 disableAPIServerFloatingIP: true tags: - openshiftClusterID=<cluster_name> network: id: <api_service_network_id> 4 externalNetwork: id: <floating_network_id> 5 identityRef: cloudName: openstack name: openstack-cloud-credentials
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: OpenStackMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 flavor: <openstack_node_machine_flavor> 4 image: filter: name: <openstack_image> 5
apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example cluster.x-k8s.io/cluster-name: <cluster_name> cluster.x-k8s.io/set-name: <machine_set_name> template: metadata: labels: test: example cluster.x-k8s.io/cluster-name: <cluster_name> cluster.x-k8s.io/set-name: <machine_set_name> node-role.kubernetes.io/<role>: "" spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: OpenStackMachineTemplate 4 name: <template_name> 5 failureDomain: <nova_availability_zone> 6
oc delete machine.machine.openshift.io <machine_name>
oc delete machine.cluster.x-k8s.io <machine_name>