Chapter 12. Managing machines with the Cluster API Important Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . The Cluster API is an upstream project that is integrated into OpenShift Container Platform as a Technology Preview for Amazon Web Services (AWS) and Google Cloud Platform (GCP) clusters. You can use the Cluster API to create and manage machine sets and machines in your OpenShift Container Platform cluster. This capability is in addition or an alternative to managing machines with the Machine API. For OpenShift Container Platform 4.11 clusters, you can use the Cluster API to perform node host provisioning management actions after the cluster installation finishes. This system enables an elastic, dynamic provisioning method on top of public or private cloud infrastructure. With the Cluster API Technology Preview, you can create compute machines and machine sets on OpenShift Container Platform clusters for supported providers. You can also explore the features that are enabled by this implementation that might not be available with the Machine API. Benefits By using the Cluster API, OpenShift Container Platform users and developers are able to realize the following advantages: The option to use upstream community Cluster API infrastructure providers which might not be supported by the Machine API. The opportunity to collaborate with third parties who maintain machine controllers for infrastructure providers. The ability to use the same set of Kubernetes tools for infrastructure management in OpenShift Container Platform. The ability to create machine sets using the Cluster API that support features that are not available with the Machine API. Limitations Using the Cluster API to manage machines is a Technology Preview feature and has the following limitations: Only AWS and GCP clusters are supported. To use this feature, you must enable the TechPreviewNoUpgrade feature set . Enabling this feature set cannot be undone and prevents minor version updates. You must create the primary resources that the Cluster API requires manually. Control plane machines cannot be managed by the Cluster API. Migration of existing machine sets created by the Machine API to Cluster API machine sets is not supported. Full feature parity with the Machine API is not available. 12.1. Cluster API architecture The OpenShift Container Platform integration of the upstream Cluster API is implemented and managed by the Cluster CAPI Operator. The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, in contrast to the Machine API, which uses the openshift-machine-api namespace. 12.1.1. The Cluster CAPI Operator The Cluster CAPI Operator is an OpenShift Container Platform Operator that maintains the lifecycle of Cluster API resources. This Operator is responsible for all administrative tasks related to deploying the Cluster API project within an OpenShift Container Platform cluster. If a cluster is configured correctly to allow the use of the Cluster API, the Cluster CAPI Operator installs the Cluster API Operator on the cluster. Note The Cluster CAPI Operator is distinct from the upstream Cluster API Operator. For more information, see the entry for the Cluster CAPI Operator in the Cluster Operators reference content. 12.1.2. Primary resources The Cluster API is comprised of the following primary resources. For the Technology Preview of this feature, you must create these resources manually in the openshift-cluster-api namespace. Cluster A fundamental unit that represents a cluster that is managed by the Cluster API. Infrastructure A provider-specific resource that defines properties that are shared by all the machine sets in the cluster, such as the region and subnets. Machine template A provider-specific template that defines the properties of the machines that a machine set creates. Machine set A group of machines. Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute needs. With the Cluster API, a machine set references a Cluster object and a provider-specific machine template. Machine A fundamental unit that describes the host for a node. The Cluster API creates machines based on the configuration in the machine template. Additional resources Cluster CAPI Operator 12.2. Sample YAML files For the Cluster API Technology Preview, you must create the primary resources that the Cluster API requires manually. The example YAML files in this section demonstrate how to make these resources work together and configure settings for the machines that they create that are appropriate for your environment. 12.2.1. Sample YAML for a Cluster API cluster resource The cluster resource defines the name and infrastructure provider for the cluster and is managed by the Cluster API. This resource has the same structure for all providers. apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: <cluster_name> 1 namespace: openshift-cluster-api spec: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <infrastructure_kind> 2 name: <cluster_name> 3 namespace: openshift-cluster-api 1 3 Specify the name of the cluster. 2 Specify the infrastructure kind for the cluster. Valid values are: AWSCluster : The cluster is running on Amazon Web Services (AWS). GCPCluster : The cluster is running on Google Cloud Platform (GCP). The remaining Cluster API resources are provider-specific. Refer to the example YAML files for your cluster: Sample YAML files for configuring Amazon Web Services clusters Sample YAML files for configuring Google Cloud Platform clusters 12.2.2. Sample YAML files for configuring Amazon Web Services clusters Some Cluster API resources are provider-specific. The example YAML files in this section show configurations for an Amazon Web Services (AWS) cluster. 12.2.2.1. Sample YAML for a Cluster API infrastructure resource on Amazon Web Services The infrastructure resource is provider-specific and defines properties that are shared by all the machine sets in the cluster, such as the region and subnets. The machine set references this resource when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSCluster 1 metadata: name: <cluster_name> 2 namespace: openshift-cluster-api spec: region: <region> 3 1 Specify the infrastructure kind for the cluster. This value must match the value for your platform. 2 Specify the name of the cluster. 3 Specify the AWS region. 12.2.2.2. Sample YAML for a Cluster API machine template resource on Amazon Web Services The machine template resource is provider-specific and defines the basic properties of the machines that a machine set creates. The machine set references this template when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 kind: AWSMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 uncompressedUserData: true iamInstanceProfile: .... instanceType: m5.large cloudInit: insecureSkipSecretsManager: true ami: id: .... subnet: filters: - name: tag:Name values: - ... additionalSecurityGroups: - filters: - name: tag:Name values: - ... 1 Specify the machine template kind. This value must match the value for your platform. 2 Specify a name for the machine template. 3 Specify the details for your environment. The values here are examples. 12.2.2.3. Sample YAML for a Cluster API machine set resource on Amazon Web Services The machine set resource defines additional properties of the machines that it creates. The machine set also references the infrastructure resource and machine template when creating machines. apiVersion: cluster.x-k8s.io/v1alpha4 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> 4 infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 kind: AWSMachineTemplate 5 name: <cluster_name> 6 1 Specify a name for the machine set. 2 4 6 Specify the name of the cluster. 3 For the Cluster API Technology Preview, the Operator can use the worker user data secret from openshift-machine-api namespace. 5 Specify the machine template kind. This value must match the value for your platform. 12.2.3. Sample YAML files for configuring Google Cloud Platform clusters Some Cluster API resources are provider-specific. The example YAML files in this section show configurations for a Google Cloud Platform (GCP) cluster. 12.2.3.1. Sample YAML for a Cluster API infrastructure resource on Google Cloud Platform The infrastructure resource is provider-specific and defines properties that are shared by all the machine sets in the cluster, such as the region and subnets. The machine set references this resource when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPCluster 1 metadata: name: <cluster_name> 2 spec: network: name: <cluster_name>-network 3 project: <project> 4 region: <region> 5 1 Specify the infrastructure kind for the cluster. This value must match the value for your platform. 2 3 Specify the name of the cluster. 4 Specify the GCP project name. 5 Specify the GCP region. 12.2.3.2. Sample YAML for a Cluster API machine template resource on Google Cloud Platform The machine template resource is provider-specific and defines the basic properties of the machines that a machine set creates. The machine set references this template when creating machines. apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 rootDeviceType: pd-ssd rootDeviceSize: 128 instanceType: n1-standard-4 image: projects/rhcos-cloud/global/images/rhcos-411-85-202203181601-0-gcp-x86-64 subnet: <cluster_name>-worker-subnet serviceAccounts: email: <service_account_email_address> scopes: - https://www.googleapis.com/auth/cloud-platform additionalLabels: kubernetes-io-cluster-<cluster_name>: owned additionalNetworkTags: - <cluster_name>-worker ipForwarding: Disabled 1 Specify the machine template kind. This value must match the value for your platform. 2 Specify a name for the machine template. 3 Specify the details for your environment. The values here are examples. 12.2.3.3. Sample YAML for a Cluster API machine set resource on Google Cloud Platform The machine set resource defines additional properties of the machines that it creates. The machine set also references the infrastructure resource and machine template when creating machines. apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: test template: metadata: labels: test: test spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> 4 infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 5 name: <machine_set_name> 6 failureDomain: <failure_domain> 7 1 6 Specify a name for the machine set. 2 4 Specify the name of the cluster. 3 For the Cluster API Technology Preview, the Operator can use the worker user data secret from openshift-machine-api namespace. 5 Specify the machine template kind. This value must match the value for your platform. 7 Specify the failure domain within the GCP region. 12.3. Creating a Cluster API machine set You can create machine sets that use the Cluster API to dynamically manage the machine compute resources for specific workloads of your choice. Prerequisites Deploy an OpenShift Container Platform cluster. Enable the use of the Cluster API. Install the OpenShift CLI ( oc ). Log in to oc as a user with cluster-admin permission. Procedure Create a YAML file that contains the cluster custom resource (CR) and is named <cluster_resource_file>.yaml . If you are not sure which value to set for the <cluster_name> parameter, you can check the value for an existing Machine API machine set in your cluster. To list the Machine API machine sets, run the following command: USD oc get machinesets -n openshift-machine-api 1 1 Specify the openshift-machine-api namespace. Example output NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m To display the contents of a specific machine set CR, run the following command: USD oc get machineset <machineset_name> \ -n openshift-machine-api \ -o yaml Example output ... template: metadata: labels: machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1 machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a ... 1 The cluster ID, which you use for the <cluster_name> parameter. Create the cluster CR by running the following command: USD oc create -f <cluster_resource_file>.yaml Verification To confirm that the cluster CR is created, run the following command: USD oc get cluster Example output NAME PHASE AGE VERSION <cluster_name> Provisioning 4h6m Create a YAML file that contains the infrastructure CR and is named <infrastructure_resource_file>.yaml . Create the infrastructure CR by running the following command: USD oc create -f <infrastructure_resource_file>.yaml Verification To confirm that the infrastructure CR is created, run the following command: USD oc get <infrastructure_kind> where <infrastructure_kind> is the value that corresponds to your platform. Example output NAME CLUSTER READY VPC BASTION IP <cluster_name> <cluster_name> true Create a YAML file that contains the machine template CR and is named <machine_template_resource_file>.yaml . Create the machine template CR by running the following command: USD oc create -f <machine_template_resource_file>.yaml Verification To confirm that the machine template CR is created, run the following command: USD oc get <machine_template_kind> where <machine_template_kind> is the value that corresponds to your platform. Example output NAME AGE <template_name> 77m Create a YAML file that contains the machine set CR and is named <machine_set_resource_file>.yaml . Create the machine set CR by running the following command: USD oc create -f <machine_set_resource_file>.yaml Verification To confirm that the machine set CR is created, run the following command: USD oc get machineset -n openshift-cluster-api 1 1 Specify the openshift-cluster-api namespace. Example output NAME CLUSTER REPLICAS READY AVAILABLE AGE VERSION <machine_set_name> <cluster_name> 1 1 1 17m When the new machine set is available, the REPLICAS and AVAILABLE values match. If the machine set is not available, wait a few minutes and run the command again. Verification To verify that the machine set is creating machines according to your desired configuration, you can review the lists of machines and nodes in the cluster. To view the list of Cluster API machines, run the following command: USD oc get machine -n openshift-cluster-api 1 1 Specify the openshift-cluster-api namespace. Example output NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_set_name>-<string_id> <cluster_name> <ip_address>.<region>.compute.internal <provider_id> Running 8m23s To view the list of nodes, run the following command: USD oc get node Example output NAME STATUS ROLES AGE VERSION <ip_address_1>.<region>.compute.internal Ready worker 5h14m v1.24.0+284d62a <ip_address_2>.<region>.compute.internal Ready master 5h19m v1.24.0+284d62a <ip_address_3>.<region>.compute.internal Ready worker 7m v1.24.0+284d62a 12.4. Troubleshooting clusters that use the Cluster API Use the information in this section to understand and recover from issues you might encounter. Generally, troubleshooting steps for problems with the Cluster API are similar to those steps for problems with the Machine API. The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, whereas the Machine API uses the openshift-machine-api namespace. When using oc commands that reference a namespace, be sure to reference the correct one. 12.4.1. CLI commands return Cluster API machines For clusters that use the Cluster API, oc commands such as oc get machine return results for Cluster API machines. Because the letter c precedes the letter m alphabetically, Cluster API machines appear in the return before Machine API machines do. To list only Machine API machines, use the fully qualified name machines.machine.openshift.io when running the oc get machine command: USD oc get machines.machine.openshift.io To list only Cluster API machines, use the fully qualified name machines.cluster.x-k8s.io when running the oc get machine command: USD oc get machines.cluster.x-k8s.io

Commands:
apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: <cluster_name> 1 namespace: openshift-cluster-api spec: infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: <infrastructure_kind> 2 name: <cluster_name> 3 namespace: openshift-cluster-api
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSCluster 1 metadata: name: <cluster_name> 2 namespace: openshift-cluster-api spec: region: <region> 3
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 kind: AWSMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 uncompressedUserData: true iamInstanceProfile: . instanceType: m5.large cloudInit: insecureSkipSecretsManager: true ami: id: . subnet: filters: - name: tag:Name values: - additionalSecurityGroups: - filters: - name: tag:Name values: -
apiVersion: cluster.x-k8s.io/v1alpha4 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: example template: metadata: labels: test: example spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> 4 infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4 kind: AWSMachineTemplate 5 name: <cluster_name> 6
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPCluster 1 metadata: name: <cluster_name> 2 spec: network: name: <cluster_name>-network 3 project: <project> 4 region: <region> 5
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 1 metadata: name: <template_name> 2 namespace: openshift-cluster-api spec: template: spec: 3 rootDeviceType: pd-ssd rootDeviceSize: 128 instanceType: n1-standard-4 image: projects/rhcos-cloud/global/images/rhcos-411-85-202203181601-0-gcp-x86-64 subnet: <cluster_name>-worker-subnet serviceAccounts: email: <service_account_email_address> scopes: - https://www.googleapis.com/auth/cloud-platform additionalLabels: kubernetes-io-cluster-<cluster_name>: owned additionalNetworkTags: - <cluster_name>-worker ipForwarding: Disabled
apiVersion: cluster.x-k8s.io/v1beta1 kind: MachineSet metadata: name: <machine_set_name> 1 namespace: openshift-cluster-api spec: clusterName: <cluster_name> 2 replicas: 1 selector: matchLabels: test: test template: metadata: labels: test: test spec: bootstrap: dataSecretName: worker-user-data 3 clusterName: <cluster_name> 4 infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: GCPMachineTemplate 5 name: <machine_set_name> 6 failureDomain: <failure_domain> 7
oc get machinesets -n openshift-machine-api 1
NAME DESIRED CURRENT READY AVAILABLE AGE agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m agl030519-vplxk-worker-us-east-1d 0 0 55m agl030519-vplxk-worker-us-east-1e 0 0 55m agl030519-vplxk-worker-us-east-1f 0 0 55m
oc get machineset <machineset_name> -n openshift-machine-api -o yaml
template: metadata: labels: machine.openshift.io/cluster-api-cluster: agl030519-vplxk 1 machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a
oc create -f <cluster_resource_file>.yaml
oc get cluster
NAME PHASE AGE VERSION <cluster_name> Provisioning 4h6m
oc create -f <infrastructure_resource_file>.yaml
oc get <infrastructure_kind>
NAME CLUSTER READY VPC BASTION IP <cluster_name> <cluster_name> true
oc create -f <machine_template_resource_file>.yaml
oc get <machine_template_kind>
NAME AGE <template_name> 77m
oc create -f <machine_set_resource_file>.yaml
oc get machineset -n openshift-cluster-api 1
NAME CLUSTER REPLICAS READY AVAILABLE AGE VERSION <machine_set_name> <cluster_name> 1 1 1 17m
oc get machine -n openshift-cluster-api 1
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION <machine_set_name>-<string_id> <cluster_name> <ip_address>.<region>.compute.internal <provider_id> Running 8m23s
oc get node
NAME STATUS ROLES AGE VERSION <ip_address_1>.<region>.compute.internal Ready worker 5h14m v1.24.0+284d62a <ip_address_2>.<region>.compute.internal Ready master 5h19m v1.24.0+284d62a <ip_address_3>.<region>.compute.internal Ready worker 7m v1.24.0+284d62a
oc get machines.machine.openshift.io
oc get machines.cluster.x-k8s.io