Chapter 1. Machine configuration overview There are times when you need to make changes to the operating systems running on OpenShift Container Platform nodes. This can include changing settings for network time service, adding kernel arguments, or configuring journaling in a specific way. Aside from a few specialized features, most changes to operating systems on OpenShift Container Platform nodes can be done by creating what are referred to as MachineConfig objects that are managed by the Machine Config Operator. For example, you can use the Machine Config Operator (MCO) and machine configs to manage update to systemd, CRI-O and kubelet, the kernel, Network Manager and other system features. Tasks in this section describe how to use features of the Machine Config Operator to configure operating system features on OpenShift Container Platform nodes. Important NetworkManager stores new network configurations to /etc/NetworkManager/system-connections/ in a key file format. Previously, NetworkManager stored new network configurations to /etc/sysconfig/network-scripts/ in the ifcfg format. Starting with RHEL 9.0, RHEL stores new network configurations at /etc/NetworkManager/system-connections/ in a key file format. The connections configurations stored to /etc/sysconfig/network-scripts/ in the old format still work uninterrupted. Modifications in existing profiles continue updating the older files. 1.1. About the Machine Config Operator OpenShift Container Platform 4.18 integrates both operating system and cluster management. Because the cluster manages its own updates, including updates to Red Hat Enterprise Linux CoreOS (RHCOS) on cluster nodes, OpenShift Container Platform provides an opinionated lifecycle management experience that simplifies the orchestration of node upgrades. OpenShift Container Platform employs three daemon sets and controllers to simplify node management. These daemon sets orchestrate operating system updates and configuration changes to the hosts by using standard Kubernetes-style constructs. They include: The machine-config-controller , which coordinates machine upgrades from the control plane. It monitors all of the cluster nodes and orchestrates their configuration updates. The machine-config-daemon daemon set, which runs on each node in the cluster and updates a machine to configuration as defined by machine config and as instructed by the MachineConfigController. When the node detects a change, it drains off its pods, applies the update, and reboots. These changes come in the form of Ignition configuration files that apply the specified machine configuration and control kubelet configuration. The update itself is delivered in a container. This process is key to the success of managing OpenShift Container Platform and RHCOS updates together. The machine-config-server daemon set, which provides the Ignition config files to control plane nodes as they join the cluster. The machine configuration is a subset of the Ignition configuration. The machine-config-daemon reads the machine configuration to see if it needs to do an OSTree update or if it must apply a series of systemd kubelet file changes, configuration changes, or other changes to the operating system or OpenShift Container Platform configuration. When you perform node management operations, you create or modify a KubeletConfig custom resource (CR). Important When changes are made to a machine configuration, the Machine Config Operator (MCO) automatically reboots all corresponding nodes in order for the changes to take effect. You can mitigate the disruption caused by some machine config changes by using a node disruption policy. See Understanding node restart behaviors after machine config changes . Alternatively, you can prevent the nodes from automatically rebooting after machine configuration changes before making the changes. Pause the autoreboot process by setting the spec.paused field to true in the corresponding machine config pool. When paused, machine configuration changes are not applied until you set the spec.paused field to false and the nodes have rebooted into the new configuration. The following modifications do not trigger a node reboot: When the MCO detects any of the following changes, it applies the update without draining or rebooting the node: Changes to the SSH key in the spec.config.passwd.users.sshAuthorizedKeys parameter of a machine config. Changes to the global pull secret or pull secret in the openshift-config namespace. Automatic rotation of the /etc/kubernetes/kubelet-ca.crt certificate authority (CA) by the Kubernetes API Server Operator. When the MCO detects changes to the /etc/containers/registries.conf file, such as adding or editing an ImageDigestMirrorSet , ImageTagMirrorSet , or ImageContentSourcePolicy object, it drains the corresponding nodes, applies the changes, and uncordons the nodes. The node drain does not happen for the following changes: The addition of a registry with the pull-from-mirror = "digest-only" parameter set for each mirror. The addition of a mirror with the pull-from-mirror = "digest-only" parameter set in a registry. The addition of items to the unqualified-search-registries list. There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called configuration drift . The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node degraded until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated. Additional resources About the OVN-Kubernetes network plugin 1.2. Machine config overview The Machine Config Operator (MCO) manages updates to systemd, CRI-O and Kubelet, the kernel, Network Manager and other system features. It also offers a MachineConfig CRD that can write configuration files onto the host (see machine-config-operator ). Understanding what MCO does and how it interacts with other components is critical to making advanced, system-level changes to an OpenShift Container Platform cluster. Here are some things you should know about MCO, machine configs, and how they are used: Machine configs are processed alphabetically, in lexicographically increasing order, of their name. The render controller uses the first machine config in the list as the base and appends the rest to the base machine config. A machine config can make a specific change to a file or service on the operating system of each system representing a pool of OpenShift Container Platform nodes. MCO applies changes to operating systems in pools of machines. All OpenShift Container Platform clusters start with worker and control plane node pools. By adding more role labels, you can configure custom pools of nodes. For example, you can set up a custom pool of worker nodes that includes particular hardware features needed by an application. However, examples in this section focus on changes to the default pool types. Important A node can have multiple labels applied that indicate its type, such as master or worker , however it can be a member of only a single machine config pool. After a machine config change, the MCO updates the affected nodes alphabetically by zone, based on the topology.kubernetes.io/zone label. If a zone has more than one node, the oldest nodes are updated first. For nodes that do not use zones, such as in bare metal deployments, the nodes are upgraded by age, with the oldest nodes updated first. The MCO updates the number of nodes as specified by the maxUnavailable field on the machine configuration pool at a time. Some machine configuration must be in place before OpenShift Container Platform is installed to disk. In most cases, this can be accomplished by creating a machine config that is injected directly into the OpenShift Container Platform installer process, instead of running as a postinstallation machine config. In other cases, you might need to do bare metal installation where you pass kernel arguments at OpenShift Container Platform installer startup, to do such things as setting per-node individual IP addresses or advanced disk partitioning. MCO manages items that are set in machine configs. Manual changes you do to your systems will not be overwritten by MCO, unless MCO is explicitly told to manage a conflicting file. In other words, MCO only makes specific updates you request, it does not claim control over the whole node. Manual changes to nodes are strongly discouraged. If you need to decommission a node and start a new one, those direct changes would be lost. MCO is only supported for writing to files in /etc and /var directories, although there are symbolic links to some directories that can be writeable by being symbolically linked to one of those areas. The /opt and /usr/local directories are examples. Ignition is the configuration format used in MachineConfigs. See the Ignition Configuration Specification v3.2.0 for details. Although Ignition config settings can be delivered directly at OpenShift Container Platform installation time, and are formatted in the same way that MCO delivers Ignition configs, MCO has no way of seeing what those original Ignition configs are. Therefore, you should wrap Ignition config settings into a machine config before deploying them. When a file managed by MCO changes outside of MCO, the Machine Config Daemon (MCD) sets the node as degraded . It will not overwrite the offending file, however, and should continue to operate in a degraded state. A key reason for using a machine config is that it will be applied when you spin up new nodes for a pool in your OpenShift Container Platform cluster. The machine-api-operator provisions a new machine and MCO configures it. MCO uses Ignition as the configuration format. OpenShift Container Platform 4.6 moved from Ignition config specification version 2 to version 3. 1.2.1. What can you change with machine configs? The kinds of components that MCO can change include: config : Create Ignition config objects (see the Ignition configuration specification ) to do things like modify files, systemd services, and other features on OpenShift Container Platform machines, including: Configuration files : Create or overwrite files in the /var or /etc directory. systemd units : Create and set the status of a systemd service or add to an existing systemd service by dropping in additional settings. users and groups : Change SSH keys in the passwd section postinstallation. Important Changing SSH keys by using a machine config is supported only for the core user. Adding new users by using a machine config is not supported. kernelArguments : Add arguments to the kernel command line when OpenShift Container Platform nodes boot. kernelType : Optionally identify a non-standard kernel to use instead of the standard kernel. Use realtime to use the RT kernel (for RAN). This is only supported on select platforms. Use the 64k-pages parameter to enable the 64k page size kernel. This setting is exclusive to machines with 64-bit ARM architectures. fips : Enable FIPS mode. FIPS should be set at installation-time setting and not a postinstallation procedure. Important To enable FIPS mode for your cluster, you must run the installation program from a Red Hat Enterprise Linux (RHEL) computer configured to operate in FIPS mode. For more information about configuring FIPS mode on RHEL, see Switching RHEL to FIPS mode . When running Red Hat Enterprise Linux (RHEL) or Red Hat Enterprise Linux CoreOS (RHCOS) booted in FIPS mode, OpenShift Container Platform core components use the RHEL cryptographic libraries that have been submitted to NIST for FIPS 140-2/140-3 Validation on only the x86_64, ppc64le, and s390x architectures. extensions : Extend RHCOS features by adding selected pre-packaged software. For this feature, available extensions include usbguard and kernel modules. Custom resources (for ContainerRuntime and Kubelet ) : Outside of machine configs, MCO manages two special custom resources for modifying CRI-O container runtime settings ( ContainerRuntime CR) and the Kubelet service ( Kubelet CR). The MCO is not the only Operator that can change operating system components on OpenShift Container Platform nodes. Other Operators can modify operating system-level features as well. One example is the Node Tuning Operator, which allows you to do node-level tuning through Tuned daemon profiles. Tasks for the MCO configuration that can be done after installation are included in the following procedures. See descriptions of RHCOS bare metal installation for system configuration tasks that must be done during or before OpenShift Container Platform installation. By default, many of the changes you make with the MCO require a reboot. The following modifications do not trigger a node reboot: When the MCO detects any of the following changes, it applies the update without draining or rebooting the node: Changes to the SSH key in the spec.config.passwd.users.sshAuthorizedKeys parameter of a machine config. Changes to the global pull secret or pull secret in the openshift-config namespace. Automatic rotation of the /etc/kubernetes/kubelet-ca.crt certificate authority (CA) by the Kubernetes API Server Operator. When the MCO detects changes to the /etc/containers/registries.conf file, such as adding or editing an ImageDigestMirrorSet , ImageTagMirrorSet , or ImageContentSourcePolicy object, it drains the corresponding nodes, applies the changes, and uncordons the nodes. The node drain does not happen for the following changes: The addition of a registry with the pull-from-mirror = "digest-only" parameter set for each mirror. The addition of a mirror with the pull-from-mirror = "digest-only" parameter set in a registry. The addition of items to the unqualified-search-registries list. In other cases, you can mitigate the disruption to your workload when the MCO makes changes by using node disruption policies . For information, see Understanding node restart behaviors after machine config changes . There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called configuration drift . The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node degraded until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated. For more information on configuration drift, see Understanding configuration drift detection . 1.2.2. Node configuration management with machine config pools Machines that run control plane components or user workloads are divided into groups based on the types of resources they handle. These groups of machines are called machine config pools (MCP). Each MCP manages a set of nodes and its corresponding machine configs. The role of the node determines which MCP it belongs to; the MCP governs nodes based on its assigned node role label. Nodes in an MCP have the same configuration; this means nodes can be scaled up and torn down in response to increased or decreased workloads. By default, there are two MCPs created by the cluster when it is installed: master and worker . Each default MCP has a defined configuration applied by the Machine Config Operator (MCO), which is responsible for managing MCPs and facilitating MCP updates. For worker nodes, you can create additional MCPs, or custom pools, to manage nodes with custom use cases that extend outside of the default node types. Custom MCPs for the control plane nodes are not supported. Custom pools are pools that inherit their configurations from the worker pool. They use any machine config targeted for the worker pool, but add the ability to deploy changes only targeted at the custom pool. Since a custom pool inherits its configuration from the worker pool, any change to the worker pool is applied to the custom pool as well. Custom pools that do not inherit their configurations from the worker pool are not supported by the MCO. Note A node can only be included in one MCP. If a node has multiple labels that correspond to several MCPs, like worker,infra , it is managed by the infra custom pool, not the worker pool. Custom pools take priority on selecting nodes to manage based on node labels; nodes that do not belong to a custom pool are managed by the worker pool. It is recommended to have a custom pool for every node role you want to manage in your cluster. For example, if you create infra nodes to handle infra workloads, it is recommended to create a custom infra MCP to group those nodes together. If you apply an infra role label to a worker node so it has the worker,infra dual label, but do not have a custom infra MCP, the MCO considers it a worker node. If you remove the worker label from a node and apply the infra label without grouping it in a custom pool, the node is not recognized by the MCO and is unmanaged by the cluster. Important Any node labeled with the infra role that is only running infra workloads is not counted toward the total number of subscriptions. The MCP managing an infra node is mutually exclusive from how the cluster determines subscription charges; tagging a node with the appropriate infra role and using taints to prevent user workloads from being scheduled on that node are the only requirements for avoiding subscription charges for infra workloads. The MCO applies updates for pools independently; for example, if there is an update that affects all pools, nodes from each pool update in parallel with each other. If you add a custom pool, nodes from that pool also attempt to update concurrently with the master and worker nodes. There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called configuration drift . The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node degraded until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated. 1.3. Understanding the Machine Config Operator node drain behavior When you use a machine config to change a system feature, such as adding new config files, modifying systemd units or kernel arguments, or updating SSH keys, the Machine Config Operator (MCO) applies those changes and ensures that each node is in the desired configuration state. After you make the changes, the MCO generates a new rendered machine config. In the majority of cases, when applying the new rendered machine config, the Operator performs the following steps on each affected node until all of the affected nodes have the updated configuration: Cordon. The MCO marks the node as not schedulable for additional workloads. Drain. The MCO terminates all running workloads on the node, causing the workloads to be rescheduled onto other nodes. Apply. The MCO writes the new configuration to the nodes as needed. Reboot. The MCO restarts the node. Uncordon. The MCO marks the node as schedulable for workloads. Throughout this process, the MCO maintains the required number of pods based on the MaxUnavailable value set in the machine config pool. Note There are conditions which can prevent the MCO from draining a node. If the MCO fails to drain a node, the Operator will be unable to reboot the node, preventing any changes made to the node through a machine config. For more information and mitigation steps, see the MCCDrainError runbook. If the MCO drains pods on the master node, note the following conditions: In single-node OpenShift clusters, the MCO skips the drain operation. The MCO does not drain static pods in order to prevent interference with services, such as etcd. Note In certain cases the nodes are not drained. For more information, see "About the Machine Config Operator." There are ways to mitigate the disruption caused by drain and reboot cycles by using node disruption policies or disabling control plane reboots. For more information, see "Understanding node restart behaviors after machine config changes" and "Disabling the Machine Config Operator from automatically rebooting." Additional resources About the Machine Config Operator Using node disruption policies to minimize disruption from machine config changes Disabling the Machine Config Operator from automatically rebooting 1.4. Understanding configuration drift detection There might be situations when the on-disk state of a node differs from what is configured in the machine config. This is known as configuration drift . For example, a cluster admin might manually modify a file, a systemd unit file, or a file permission that was configured through a machine config. This causes configuration drift. Configuration drift can cause problems between nodes in a Machine Config Pool or when the machine configs are updated. The Machine Config Operator (MCO) uses the Machine Config Daemon (MCD) to check nodes for configuration drift on a regular basis. If detected, the MCO sets the node and the machine config pool (MCP) to Degraded and reports the error. A degraded node is online and operational, but, it cannot be updated. The MCD performs configuration drift detection upon each of the following conditions: When a node boots. After any of the files (Ignition files and systemd drop-in units) specified in the machine config are modified outside of the machine config. Before a new machine config is applied. Note If you apply a new machine config to the nodes, the MCD temporarily shuts down configuration drift detection. This shutdown is needed because the new machine config necessarily differs from the machine config on the nodes. After the new machine config is applied, the MCD restarts detecting configuration drift using the new machine config. When performing configuration drift detection, the MCD validates that the file contents and permissions fully match what the currently-applied machine config specifies. Typically, the MCD detects configuration drift in less than a second after the detection is triggered. If the MCD detects configuration drift, the MCD performs the following tasks: Emits an error to the console logs Emits a Kubernetes event Stops further detection on the node Sets the node and MCP to degraded You can check if you have a degraded node by listing the MCPs: USD oc get mcp worker If you have a degraded MCP, the DEGRADEDMACHINECOUNT field is non-zero, similar to the following output: Example output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-404caf3180818d8ac1f50c32f14b57c3 False True True 2 1 1 1 5h51m You can determine if the problem is caused by configuration drift by examining the machine config pool: USD oc describe mcp worker Example output ... Last Transition Time: 2021-12-20T18:54:00Z Message: Node ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4 is reporting: "content mismatch for file \"/etc/mco-test-file\"" 1 Reason: 1 nodes are reporting degraded status on sync Status: True Type: NodeDegraded 2 ... 1 This message shows that a node's /etc/mco-test-file file, which was added by the machine config, has changed outside of the machine config. 2 The state of the node is NodeDegraded . Or, if you know which node is degraded, examine that node: USD oc describe node/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4 Example output ... Annotations: cloud.network.openshift.io/egress-ipconfig: [{"interface":"nic0","ifaddr":{"ipv4":"10.0.128.0/17"},"capacity":{"ip":10}}] csi.volume.kubernetes.io/nodeid: {"pd.csi.storage.gke.io":"projects/openshift-gce-devel-ci/zones/us-central1-a/instances/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4"} machine.openshift.io/machine: openshift-machine-api/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4 machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable machineconfiguration.openshift.io/currentConfig: rendered-worker-67bd55d0b02b0f659aef33680693a9f9 machineconfiguration.openshift.io/desiredConfig: rendered-worker-67bd55d0b02b0f659aef33680693a9f9 machineconfiguration.openshift.io/reason: content mismatch for file "/etc/mco-test-file" 1 machineconfiguration.openshift.io/state: Degraded 2 ... 1 The error message indicating that configuration drift was detected between the node and the listed machine config. Here the error message indicates that the contents of the /etc/mco-test-file , which was added by the machine config, has changed outside of the machine config. 2 The state of the node is Degraded . You can correct configuration drift and return the node to the Ready state by performing one of the following remediations: Ensure that the contents and file permissions of the files on the node match what is configured in the machine config. You can manually rewrite the file contents or change the file permissions. Generate a force file on the degraded node. The force file causes the MCD to bypass the usual configuration drift detection and reapplies the current machine config. Note Generating a force file on a node causes that node to reboot. 1.5. Checking machine config pool status To see the status of the Machine Config Operator (MCO), its sub-components, and the resources it manages, use the following oc commands: Procedure To see the number of MCO-managed nodes available on your cluster for each machine config pool (MCP), run the following command: USD oc get machineconfigpool Example output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-06c9c4... True False False 3 3 3 0 4h42m worker rendered-worker-f4b64... False True False 3 2 2 0 4h42m where: UPDATED The True status indicates that the MCO has applied the current machine config to the nodes in that MCP. The current machine config is specified in the STATUS field in the oc get mcp output. The False status indicates a node in the MCP is updating. UPDATING The True status indicates that the MCO is applying the desired machine config, as specified in the MachineConfigPool custom resource, to at least one of the nodes in that MCP. The desired machine config is the new, edited machine config. Nodes that are updating might not be available for scheduling. The False status indicates that all nodes in the MCP are updated. DEGRADED A True status indicates the MCO is blocked from applying the current or desired machine config to at least one of the nodes in that MCP, or the configuration is failing. Nodes that are degraded might not be available for scheduling. A False status indicates that all nodes in the MCP are ready. MACHINECOUNT Indicates the total number of machines in that MCP. READYMACHINECOUNT Indicates the total number of machines in that MCP that are ready for scheduling. UPDATEDMACHINECOUNT Indicates the total number of machines in that MCP that have the current machine config. DEGRADEDMACHINECOUNT Indicates the total number of machines in that MCP that are marked as degraded or unreconcilable. In the output, there are three control plane (master) nodes and three worker nodes. The control plane MCP and the associated nodes are updated to the current machine config. The nodes in the worker MCP are being updated to the desired machine config. Two of the nodes in the worker MCP are updated and one is still updating, as indicated by the UPDATEDMACHINECOUNT being 2 . There are no issues, as indicated by the DEGRADEDMACHINECOUNT being 0 and DEGRADED being False . While the nodes in the MCP are updating, the machine config listed under CONFIG is the current machine config, which the MCP is being updated from. When the update is complete, the listed machine config is the desired machine config, which the MCP was updated to. Note If a node is being cordoned, that node is not included in the READYMACHINECOUNT , but is included in the MACHINECOUNT . Also, the MCP status is set to UPDATING . Because the node has the current machine config, it is counted in the UPDATEDMACHINECOUNT total: Example output NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-06c9c4... True False False 3 3 3 0 4h42m worker rendered-worker-c1b41a... False True False 3 2 3 0 4h42m To check the status of the nodes in an MCP by examining the MachineConfigPool custom resource, run the following command: : USD oc describe mcp worker Example output ... Degraded Machine Count: 0 Machine Count: 3 Observed Generation: 2 Ready Machine Count: 3 Unavailable Machine Count: 0 Updated Machine Count: 3 Events: <none> Note If a node is being cordoned, the node is not included in the Ready Machine Count . It is included in the Unavailable Machine Count : Example output ... Degraded Machine Count: 0 Machine Count: 3 Observed Generation: 2 Ready Machine Count: 2 Unavailable Machine Count: 1 Updated Machine Count: 3 To see each existing MachineConfig object, run the following command: USD oc get machineconfigs Example output NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m 00-worker 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m 01-master-container-runtime 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m 01-master-kubelet 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m ... rendered-master-dde... 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m rendered-worker-fde... 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m Note that the MachineConfig objects listed as rendered are not meant to be changed or deleted. To view the contents of a particular machine config (in this case, 01-master-kubelet ), run the following command: USD oc describe machineconfigs 01-master-kubelet The output from the command shows that this MachineConfig object contains both configuration files ( cloud.conf and kubelet.conf ) and a systemd service (Kubernetes Kubelet): Example output Name: 01-master-kubelet ... Spec: Config: Ignition: Version: 3.2.0 Storage: Files: Contents: Source: data:, Mode: 420 Overwrite: true Path: /etc/kubernetes/cloud.conf Contents: Source: data:,kind%3A%20KubeletConfiguration%0AapiVersion%3A%20kubelet.config.k8s.io%2Fv1beta1%0Aauthentication%3A%0A%20%20x509%3A%0A%20%20%20%20clientCAFile%3A%20%2Fetc%2Fkubernetes%2Fkubelet-ca.crt%0A%20%20anonymous... Mode: 420 Overwrite: true Path: /etc/kubernetes/kubelet.conf Systemd: Units: Contents: [Unit] Description=Kubernetes Kubelet Wants=rpc-statd.service network-online.target crio.service After=network-online.target crio.service ExecStart=/usr/bin/hyperkube \ kubelet \ --config=/etc/kubernetes/kubelet.conf \ ... If something goes wrong with a machine config that you apply, you can always back out that change. For example, if you had run oc create -f ./myconfig.yaml to apply a machine config, you could remove that machine config by running the following command: USD oc delete -f ./myconfig.yaml If that was the only problem, the nodes in the affected pool should return to a non-degraded state. This actually causes the rendered configuration to roll back to its previously rendered state. If you add your own machine configs to your cluster, you can use the commands shown in the example to check their status and the related status of the pool to which they are applied. 1.6. About checking machine config node status If you make changes to the default control plane or worker nodes in your cluster, for example by using a machine config or kubelet config, you can get detailed information about the progress of the node updates by using the machine config nodes custom resource. Any change that results in a new machine config triggers the node update process. You can view this detailed information for nodes in the control plane and worker machine config pools that were created upon OpenShift Container Platform installation. You cannot use custom machine config pools. The MachineConfigNode custom resource allows you to monitor the progress of individual node updates as they move through the upgrade phases. This information can be helpful with troubleshooting if one of the nodes has an issue during the update. The custom resource reports where in the update process the node is at the moment, the phases that have completed, and the phases that are remaining. Important The machine config node custom resource is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . The node update process consists of the following phases and subphases that are tracked by the machine config node custom resource, explained with more detail later in this section: Update Prepared . The MCO stops the configuration drift monitoring process and verifies that the newly-created machine config can be applied to a node. UpdateCompatible Update Executed . The MCO cordons and drains the node and applies the new machine config to the node's files and operating system, as needed. It contains the following sub-phases: Cordoned Drained AppliedFilesAndOS PinnedImageSetsProgressing The MCO is performing the steps needed to pin and pre-load container images. PinnedImageSetsDegraded The pinned image process failed. You can view the reason for the failure by using the oc describe machineconfignode command, as described later in this section. Update Post update action . The MCO reboots the node or reloads CRI-O, as needed. RebootedNode ReloadedCRIO Update Complete . The MCO uncordons the nodes, updates the node state to the cluster, and resumes producing node metrics. Updated Uncordoned Resumed . The MCO restarts the config drift monitor process and the node returns to operational state. As the update moves through these phases, you can query the MachineConfigNode custom resource, which reports one of the following conditions for each phase: True . The phase is complete on that node or the MCO has started that phase on that node. False . The phase is either being executed or will not be executed on that node. Unknown . The phase is either being executed on that node or has an error. If the phase has an error, you can use the oc describe machineconfignodes command for more information, as described later in this section. For example, consider a cluster with a newly-created machine config: USD oc get machineconfig Example output # ... rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 6d15h rendered-master-a386c2d1550b927d274054124f58be68 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 7m26s # ... rendered-worker-01f27f752eb84eba917450e43636b210 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 6d15h 1 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 7m26s 2 # ... 1 The current machine config for the control plane and worker nodes. 2 A newly-created machine config that is being applied to the control plane and worker nodes. You can watch as the nodes are updated with the new machine config: USD oc get machineconfignodes Example output NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED ci-ln-ds73n5t-72292-9xsm9-master-0 master rendered-master-a386c2d1550b927d274054124f58be68 rendered-master-a386c2d1550b927d274054124f58be68 True ci-ln-ds73n5t-72292-9xsm9-master-1 master rendered-master-a386c2d1550b927d274054124f58be68 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 False ci-ln-ds73n5t-72292-9xsm9-master-2 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True ci-ln-ds73n5t-72292-9xsm9-worker-a-2d8tz worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 True 1 ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-01f27f752eb84eba917450e43636b210 False 2 ci-ln-ds73n5t-72292-9xsm9-worker-c-t227w worker rendered-worker-01f27f752eb84eba917450e43636b210 rendered-worker-01f27f752eb84eba917450e43636b210 True 3 1 This node has been updated. The new machine config, rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 , is shown as the desired and current machine configs. 2 This node is currently being updated to the new machine config. The and new machine configs are shown as the desired and current machine configs, respectively. 3 This node has not yet been updated to the new machine config. The machine config is shown as the desired and current machine configs. Table 1.1. Basic machine config node fields Field Meaning NAME The name of the node. POOLNAME The name of the machine config pool associated with that node. DESIREDCONFIG The name of the new machine config that the node updates to. CURRENTCONFIG The name of the current machine configuration on that node. UPDATED Indicates if the node has been updated with one of the following conditions: If False , the node is being updated to the new machine configuration shown in the DESIREDCONFIG field. If True , and the CURRENTCONFIG matches the new machine configuration shown in the DESIREDCONFIG field, the node has been updated. If True , and the CURRENTCONFIG matches the old machine configuration shown in the DESIREDCONFIG field, that node has not been updated yet. You can use the -o wide flag to display additional information about the updates: USD oc get machineconfignodes -o wide Example output USD oc get machineconfignode -o wide NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED UPDATEPREPARED UPDATEEXECUTED UPDATEPOSTACTIONCOMPLETE UPDATECOMPLETE RESUMED UPDATECOMPATIBLE UPDATEDFILESANDOS CORDONEDNODE DRAINEDNODE REBOOTEDNODE RELOADEDCRIO UNCORDONEDNODE ci-ln-ds73n5t-72292-9xsm9-master-0 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-master-1 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-master-2 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-worker-a-2d8tz worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-01f27f752eb84eba917450e43636b210 False True True Unknown False False True True True True Unknown False False ci-ln-ds73n5t-72292-9xsm9-worker-c-t227w worker rendered-worker-01f27f752eb84eba917450e43636b210 rendered-worker-01f27f752eb84eba917450e43636b210 True False False False False False False False False False False False False In addition to the fields defined in the table, the -o wide output displays the following fields: Table 1.2. Machine config node fields in the -o wide output Phase Name Definition UPDATEPREPARED Indicates if the MCO is preparing to update the node. UPDATEEXECUTED Indicates if the MCO has completed the body of the update on the node. UPDATEPOSTACTIONCOMPLETE Indicates if the MCO has executed the post-update actions on the node. UPDATECOMPLETE Indicates if the MCO has completed the update on the node. RESUMED Indicates if the node has resumed normal processes. UPDATECOMPATIBLE Indicates if the MCO has determined it can execute the update on the node. UPDATEDFILESANDOS Indicates if the MCO has updated the node files and operating system. CORDONEDNODE Indicates if the MCO has marked the node as not schedulable. DRAINEDNODE Indicates if the MCO has drained the node. REBOOTEDNODE Indicates if the MCO has restarted the node. RELOADEDCRIO Indicates if the MCO has restarted the CRI-O service. UNCORDONEDNODE Indicates if the MCO has marked the node as schedulable. For more details on the update status, you can use the oc describe machineconfignode command: USD oc describe machineconfignode/<machine_config_node_name> -o yaml Example output Name: <machine_config_node_name> 1 Namespace: Labels: <none> Annotations: <none> API Version: machineconfiguration.openshift.io/v1alpha1 Kind: MachineConfigNode Metadata: Creation Timestamp: 2023-10-17T13:08:58Z Generation: 1 Resource Version: 49443 UID: 4bd758ab-2187-413c-ac42-882e61761b1d Spec: Node Ref: Name: <node_name> Pool: Name: worker ConfigVersion: Desired: rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 2 Status: Conditions: Last Transition Time: 2025-01-14T17:01:16Z Message: Node ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd needs an update Reason: Updated Status: False Type: Updated Last Transition Time: 2025-01-14T17:01:18Z Message: Update is Compatible. Reason: UpdateCompatible Status: True Type: UpdatePrepared Last Transition Time: 2025-01-14T17:04:08Z Message: Updated the Files and OS on disk as a part of the in progress phase Reason: AppliedFilesAndOS Status: True Type: UpdateExecuted Last Transition Time: 2025-01-14T17:04:08Z Message: Node will reboot into config rendered-worker-db01b33f959e5645a721da50a6db1fbb Reason: RebootedNode Status: Unknown Type: UpdatePostActionComplete Last Transition Time: 2025-01-14T16:04:27Z Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: UnCordoned Node as part of completing upgrade phase Reason: Uncordoned Status: False Type: UpdateComplete Last Transition Time: 2025-01-14T16:04:27Z Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: In desired config rendered-worker-01f27f752eb84eba917450e43636b210. Resumed normal operations. Reason: Resumed Status: False Type: Resumed Last Transition Time: 2025-01-14T17:01:18Z Message: Update Compatible. Post Cfg Actions []: Drain Required: true Reason: UpdatePreparedUpdateCompatible Status: True Type: UpdateCompatible Last Transition Time: 2025-01-14T17:03:57Z Message: Drained node. The drain is complete as the desired drainer matches current drainer: drain-rendered-worker-db01b33f959e5645a721da50a6db1fbb Reason: UpdateExecutedDrained Status: True Type: Drained Last Transition Time: 2025-01-14T17:04:08Z Message: Applied files and new OS config to node. OS did not need an update. SSH Keys did not need an update Reason: UpdateExecutedAppliedFilesAndOS Status: True Type: AppliedFilesAndOS Last Transition Time: 2025-01-14T17:01:23Z Message: Cordoned node. The node is reporting Unschedulable = true Reason: UpdateExecutedCordoned Status: True Type: Cordoned Last Transition Time: 2025-01-14T17:04:08Z Message: Upgrade requires a reboot. Currently doing this as the post update action. Reason: UpdatePostActionCompleteRebootedNode Status: Unknown Type: RebootedNode Last Transition Time: 2025-01-14T15:30:57Z Message: This node has not yet entered the ReloadedCRIO phase Reason: NotYetOccured Status: False Type: ReloadedCRIO Last Transition Time: 2025-01-14T16:04:27Z Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: UnCordoned node. The node is reporting Unschedulable = false Reason: UpdateCompleteUncordoned Status: False Type: Uncordoned Last Transition Time: 2025-01-14T16:04:07Z Message: All is good Reason: AsExpected Status: False Type: PinnedImageSetsDegraded Last Transition Time: 2025-01-14T16:04:07Z Message: All pinned image sets complete Reason: AsExpected Status: False Type: PinnedImageSetsProgressing Config Version: Current: rendered-worker-01f27f752eb84eba917450e43636b210 3 Desired: rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 Observed Generation: 6 # ... 1 The MachineConfigNode object name. 2 The new machine configuration. This field updates after the MCO validates the machine config in the UPDATEPREPARED phase, then the status adds the new configuration. 3 The current machine config on the node. 1.6.1. Checking machine config node status During the update of a machine config pool (MCP), you can monitor the progress of all control plane and worker nodes in your cluster by using the oc get machineconfignodes and oc describe machineconfignodes commands. These commands provide information that can be helpful if issues arise during the update and you need to troubleshoot a node. You cannot use these commands with custom machine config pools. For more information on the meaning of these fields, see "About checking machine config node status." Prerequisites You enabled the required Technology Preview features for your cluster by editing the FeatureGate CR named cluster : USD oc edit featuregate cluster Example FeatureGate CR apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: name: cluster spec: featureSet: TechPreviewNoUpgrade Warning Enabling the TechPreviewNoUpgrade feature set on your cluster cannot be undone and prevents minor version updates. This feature set allows you to enable these Technology Preview features on test clusters, where you can fully test them. Do not enable this feature set on production clusters. After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied. Procedure View the update status of all nodes in the cluster, including the current and desired machine configurations, by running the following command: USD oc get machineconfignodes Example output NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED ci-ln-mdb23yt-72292-kzdsg-master-0 master rendered-master-f21b093d20f68a7c06f922ed3ea5fbc8 rendered-master-1abc053eec29e6c945670f39d6dc8afa False ci-ln-mdb23yt-72292-kzdsg-master-1 master rendered-master-1abc053eec29e6c945670f39d6dc8afa rendered-master-1abc053eec29e6c945670f39d6dc8afa True ci-ln-mdb23yt-72292-kzdsg-master-2 master rendered-master-1abc053eec29e6c945670f39d6dc8afa rendered-master-1abc053eec29e6c945670f39d6dc8afa True ci-ln-mdb23yt-72292-kzdsg-worker-a-gfqjr worker rendered-worker-d0130cd74e9e576d7ba78ce166272bfb rendered-worker-8f61bf839898a4487c3b5263a430e94a False ci-ln-mdb23yt-72292-kzdsg-worker-b-gknq4 worker rendered-worker-8f61bf839898a4487c3b5263a430e94a rendered-worker-8f61bf839898a4487c3b5263a430e94a True ci-ln-mdb23yt-72292-kzdsg-worker-c-mffrx worker rendered-worker-8f61bf839898a4487c3b5263a430e94a rendered-worker-8f61bf839898a4487c3b5263a430e94a True View of all machine config node status fields for the nodes in your cluster by running the following command: USD oc get machineconfignodes -o wide Example output NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED UPDATEPREPARED UPDATEEXECUTED UPDATEPOSTACTIONCOMPLETE UPDATECOMPLETE RESUMED UPDATECOMPATIBLE UPDATEDFILESANDOS CORDONEDNODE DRAINEDNODE REBOOTEDNODE RELOADEDCRIO UNCORDONEDNODE ci-ln-g6dr34b-72292-g9btv-master-0 master rendered-master-d4e122320b351cdbe1df59ddb63ddcfc rendered-master-6f2064fcb36d2a914de5b0c660dc49ff False True Unknown False False False True Unknown False False False False False ci-ln-g6dr34b-72292-g9btv-master-1 master rendered-master-6f2064fcb36d2a914de5b0c660dc49ff rendered-master-6f2064fcb36d2a914de5b0c660dc49ff True False False False False False False False False False False False False ci-ln-g6dr34b-72292-g9btv-master-2 master rendered-master-6f2064fcb36d2a914de5b0c660dc49ff rendered-master-6f2064fcb36d2a914de5b0c660dc49ff True False False False False False False False False False False False False ci-ln-g6dr34b-72292-g9btv-worker-a-sjh5r worker rendered-worker-671b88c8c569fa3f60dc1a27cf9c91f2 rendered-worker-d5534cb730e5e108905fc285c2a42b6c False True Unknown False False False True Unknown False False False False False ci-ln-g6dr34b-72292-g9btv-worker-b-xthbz worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-d5534cb730e5e108905fc285c2a42b6c True False False False False False False False False False False False False ci-ln-g6dr34b-72292-g9btv-worker-c-gnpd6 worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-d5534cb730e5e108905fc285c2a42b6c True False False False False False False False False False False False False Check the update status of nodes in a specific machine config pool by running the following command: USD oc get machineconfignodes USD(oc get machineconfignodes -o json | jq -r '.items[]|select(.spec.pool.name=="<pool_name>")|.metadata.name') 1 Example output NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED ci-ln-g6dr34b-72292-g9btv-worker-a-sjh5r worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-d5534cb730e5e108905fc285c2a42b6c True ci-ln-g6dr34b-72292-g9btv-worker-b-xthbz worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-faf6b50218a8bbce21f1370866283de5 False ci-ln-g6dr34b-72292-g9btv-worker-c-gnpd6 worker rendered-worker-faf6b50218a8bbce21f1370866283de5 rendered-worker-faf6b50218a8bbce21f1370866283de5 True Check the update status of an individual node by running the following command: USD oc describe machineconfignode/<node_name> Example output Name: <node_name> Namespace: Labels: <none> Annotations: <none> API Version: machineconfiguration.openshift.io/v1alpha1 Kind: MachineConfigNode Metadata: Creation Timestamp: 2023-10-17T13:08:58Z Generation: 1 Resource Version: 49443 UID: 4bd758ab-2187-413c-ac42-882e61761b1d Spec: Node Ref: Name: <node_name> Pool: Name: master ConfigVersion: Desired: rendered-worker-823ff8dc2b33bf444709ed7cd2b9855b Status: # ... Message: Drained node. The drain is complete as the desired drainer matches current drainer: drain-rendered-worker-01f27f752eb84eba917450e43636b210 Reason: UpdateExecutedDrained Status: True Type: Drained Last Transition Time: 2025-01-14T15:45:55Z # ... Config Version: Current: rendered-master-8110974a5cea69dff5b263237b58abd8 Desired: rendered-worker-823ff8dc2b33bf444709ed7cd2b9855b Observed Generation: 6 # ... Additional resources For more information about feature gates, see Enabling feature sets using the web console . 1.7. Understanding Machine Config Operator certificates Machine Config Operator certificates are used to secure connections between the Red Hat Enterprise Linux CoreOS (RHCOS) nodes and the Machine Config Server. For more information, see Machine Config Operator certificates . 1.7.1. Viewing and interacting with certificates The following certificates are handled in the cluster by the Machine Config Controller (MCC) and can be found in the ControllerConfig resource: /etc/kubernetes/kubelet-ca.crt /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt The MCC also handles the image registry certificates and its associated user bundle certificate. You can get information about the listed certificates, including the underyling bundle the certificate comes from, and the signing and subject data. Prerequisites This procedure contains optional steps that require that the python-yq RPM package is installed. Procedure Get detailed certificate information by running the following command: USD oc get controllerconfig/machine-config-controller -o yaml | yq -y '.status.controllerCertificates' Example output - bundleFile: KubeAPIServerServingCAData notAfter: '2034-10-23T13:13:02Z' notBefore: '2024-10-25T13:13:02Z' signer: CN=admin-kubeconfig-signer,OU=openshift subject: CN=admin-kubeconfig-signer,OU=openshift - bundleFile: KubeAPIServerServingCAData notAfter: '2024-10-26T13:13:05Z' notBefore: '2024-10-25T13:27:14Z' signer: CN=kubelet-signer,OU=openshift subject: CN=kube-csr-signer_@1729862835 - bundleFile: KubeAPIServerServingCAData notAfter: '2024-10-26T13:13:05Z' notBefore: '2024-10-25T13:13:05Z' signer: CN=kubelet-signer,OU=openshift subject: CN=kubelet-signer,OU=openshift # ... Get a simpler version of the information found in the ControllerConfig resource by checking the machine config pool status using the following command: USD oc get mcp master -o yaml | yq -y '.status.certExpirys' Example output - bundle: KubeAPIServerServingCAData expiry: '2034-10-23T13:13:02Z' subject: CN=admin-kubeconfig-signer,OU=openshift - bundle: KubeAPIServerServingCAData expiry: '2024-10-26T13:13:05Z' subject: CN=kube-csr-signer_@1729862835 - bundle: KubeAPIServerServingCAData expiry: '2024-10-26T13:13:05Z' subject: CN=kubelet-signer,OU=openshift - bundle: KubeAPIServerServingCAData expiry: '2025-10-25T13:13:05Z' subject: CN=kube-apiserver-to-kubelet-signer,OU=openshift # ... This method is meant for OpenShift Container Platform applications that already consume machine config pool information. Check which image registry certificates are on the nodes: Log in to a node: USD oc debug node/<node_name> Set /host as the root directory within the debug shell: sh-5.1# chroot /host Look at the contents of the /etc/docker/cert.d directory: sh-5.1# ls /etc/docker/certs.d Example output image-registry.openshift-image-registry.svc.cluster.local:5000 image-registry.openshift-image-registry.svc:5000

Commands:
oc get mcp worker
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE worker rendered-worker-404caf3180818d8ac1f50c32f14b57c3 False True True 2 1 1 1 5h51m
oc describe mcp worker
Last Transition Time: 2021-12-20T18:54:00Z Message: Node ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4 is reporting: "content mismatch for file \"/etc/mco-test-file\"" 1 Reason: 1 nodes are reporting degraded status on sync Status: True Type: NodeDegraded 2
oc describe node/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4
Annotations: cloud.network.openshift.io/egress-ipconfig: [{"interface":"nic0","ifaddr":{"ipv4":"10.0.128.0/17"},"capacity":{"ip":10}}] csi.volume.kubernetes.io/nodeid: {"pd.csi.storage.gke.io":"projects/openshift-gce-devel-ci/zones/us-central1-a/instances/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4"} machine.openshift.io/machine: openshift-machine-api/ci-ln-j4h8nkb-72292-pxqxz-worker-a-fjks4 machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable machineconfiguration.openshift.io/currentConfig: rendered-worker-67bd55d0b02b0f659aef33680693a9f9 machineconfiguration.openshift.io/desiredConfig: rendered-worker-67bd55d0b02b0f659aef33680693a9f9 machineconfiguration.openshift.io/reason: content mismatch for file "/etc/mco-test-file" 1 machineconfiguration.openshift.io/state: Degraded 2
oc get machineconfigpool
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-06c9c4... True False False 3 3 3 0 4h42m worker rendered-worker-f4b64... False True False 3 2 2 0 4h42m
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-06c9c4... True False False 3 3 3 0 4h42m worker rendered-worker-c1b41a... False True False 3 2 3 0 4h42m
oc describe mcp worker
Degraded Machine Count: 0 Machine Count: 3 Observed Generation: 2 Ready Machine Count: 3 Unavailable Machine Count: 0 Updated Machine Count: 3 Events: <none>
Degraded Machine Count: 0 Machine Count: 3 Observed Generation: 2 Ready Machine Count: 2 Unavailable Machine Count: 1 Updated Machine Count: 3
oc get machineconfigs
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m 00-worker 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m 01-master-container-runtime 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m 01-master-kubelet 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m rendered-master-dde... 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m rendered-worker-fde... 2c9371fbb673b97a6fe8b1c52... 3.2.0 5h18m
oc describe machineconfigs 01-master-kubelet
Name: 01-master-kubelet Spec: Config: Ignition: Version: 3.2.0 Storage: Files: Contents: Source: data:, Mode: 420 Overwrite: true Path: /etc/kubernetes/cloud.conf Contents: Source: data:,kind%3A%20KubeletConfiguration%0AapiVersion%3A%20kubelet.config.k8s.io%2Fv1beta1%0Aauthentication%3A%0A%20%20x509%3A%0A%20%20%20%20clientCAFile%3A%20%2Fetc%2Fkubernetes%2Fkubelet-ca.crt%0A%20%20anonymous Mode: 420 Overwrite: true Path: /etc/kubernetes/kubelet.conf Systemd: Units: Contents: [Unit] Description=Kubernetes Kubelet Wants=rpc-statd.service network-online.target crio.service After=network-online.target crio.service ExecStart=/usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf \
oc delete -f ./myconfig.yaml
oc get machineconfig
rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 6d15h rendered-master-a386c2d1550b927d274054124f58be68 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 7m26s rendered-worker-01f27f752eb84eba917450e43636b210 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 6d15h 1 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 c00e2c941bc6e236b50e0bf3988e6c790cf2bbb2 3.4.0 7m26s 2
oc get machineconfignodes
NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED ci-ln-ds73n5t-72292-9xsm9-master-0 master rendered-master-a386c2d1550b927d274054124f58be68 rendered-master-a386c2d1550b927d274054124f58be68 True ci-ln-ds73n5t-72292-9xsm9-master-1 master rendered-master-a386c2d1550b927d274054124f58be68 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 False ci-ln-ds73n5t-72292-9xsm9-master-2 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True ci-ln-ds73n5t-72292-9xsm9-worker-a-2d8tz worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 True 1 ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-01f27f752eb84eba917450e43636b210 False 2 ci-ln-ds73n5t-72292-9xsm9-worker-c-t227w worker rendered-worker-01f27f752eb84eba917450e43636b210 rendered-worker-01f27f752eb84eba917450e43636b210 True 3
oc get machineconfignodes -o wide
oc get machineconfignode -o wide NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED UPDATEPREPARED UPDATEEXECUTED UPDATEPOSTACTIONCOMPLETE UPDATECOMPLETE RESUMED UPDATECOMPATIBLE UPDATEDFILESANDOS CORDONEDNODE DRAINEDNODE REBOOTEDNODE RELOADEDCRIO UNCORDONEDNODE ci-ln-ds73n5t-72292-9xsm9-master-0 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-master-1 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-master-2 master rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 rendered-master-23cf200e4ee97daa6e39fdce24c9fb67 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-worker-a-2d8tz worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 True False False False False False False False False False False False False ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd worker rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 rendered-worker-01f27f752eb84eba917450e43636b210 False True True Unknown False False True True True True Unknown False False ci-ln-ds73n5t-72292-9xsm9-worker-c-t227w worker rendered-worker-01f27f752eb84eba917450e43636b210 rendered-worker-01f27f752eb84eba917450e43636b210 True False False False False False False False False False False False False
oc describe machineconfignode/<machine_config_node_name> -o yaml
Name: <machine_config_node_name> 1 Namespace: Labels: <none> Annotations: <none> API Version: machineconfiguration.openshift.io/v1alpha1 Kind: MachineConfigNode Metadata: Creation Timestamp: 2023-10-17T13:08:58Z Generation: 1 Resource Version: 49443 UID: 4bd758ab-2187-413c-ac42-882e61761b1d Spec: Node Ref: Name: <node_name> Pool: Name: worker ConfigVersion: Desired: rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 2 Status: Conditions: Last Transition Time: 2025-01-14T17:01:16Z Message: Node ci-ln-ds73n5t-72292-9xsm9-worker-b-gw5sd needs an update Reason: Updated Status: False Type: Updated Last Transition Time: 2025-01-14T17:01:18Z Message: Update is Compatible. Reason: UpdateCompatible Status: True Type: UpdatePrepared Last Transition Time: 2025-01-14T17:04:08Z Message: Updated the Files and OS on disk as a part of the in progress phase Reason: AppliedFilesAndOS Status: True Type: UpdateExecuted Last Transition Time: 2025-01-14T17:04:08Z Message: Node will reboot into config rendered-worker-db01b33f959e5645a721da50a6db1fbb Reason: RebootedNode Status: Unknown Type: UpdatePostActionComplete Last Transition Time: 2025-01-14T16:04:27Z Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: UnCordoned Node as part of completing upgrade phase Reason: Uncordoned Status: False Type: UpdateComplete Last Transition Time: 2025-01-14T16:04:27Z Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: In desired config rendered-worker-01f27f752eb84eba917450e43636b210. Resumed normal operations. Reason: Resumed Status: False Type: Resumed Last Transition Time: 2025-01-14T17:01:18Z Message: Update Compatible. Post Cfg Actions []: Drain Required: true Reason: UpdatePreparedUpdateCompatible Status: True Type: UpdateCompatible Last Transition Time: 2025-01-14T17:03:57Z Message: Drained node. The drain is complete as the desired drainer matches current drainer: drain-rendered-worker-db01b33f959e5645a721da50a6db1fbb Reason: UpdateExecutedDrained Status: True Type: Drained Last Transition Time: 2025-01-14T17:04:08Z Message: Applied files and new OS config to node. OS did not need an update. SSH Keys did not need an update Reason: UpdateExecutedAppliedFilesAndOS Status: True Type: AppliedFilesAndOS Last Transition Time: 2025-01-14T17:01:23Z Message: Cordoned node. The node is reporting Unschedulable = true Reason: UpdateExecutedCordoned Status: True Type: Cordoned Last Transition Time: 2025-01-14T17:04:08Z Message: Upgrade requires a reboot. Currently doing this as the post update action. Reason: UpdatePostActionCompleteRebootedNode Status: Unknown Type: RebootedNode Last Transition Time: 2025-01-14T15:30:57Z Message: This node has not yet entered the ReloadedCRIO phase Reason: NotYetOccured Status: False Type: ReloadedCRIO Last Transition Time: 2025-01-14T16:04:27Z Message: Action during update to rendered-worker-f351f6947f15cd0380514f4b1c89f8f2: UnCordoned node. The node is reporting Unschedulable = false Reason: UpdateCompleteUncordoned Status: False Type: Uncordoned Last Transition Time: 2025-01-14T16:04:07Z Message: All is good Reason: AsExpected Status: False Type: PinnedImageSetsDegraded Last Transition Time: 2025-01-14T16:04:07Z Message: All pinned image sets complete Reason: AsExpected Status: False Type: PinnedImageSetsProgressing Config Version: Current: rendered-worker-01f27f752eb84eba917450e43636b210 3 Desired: rendered-worker-f351f6947f15cd0380514f4b1c89f8f2 Observed Generation: 6
oc edit featuregate cluster
apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: name: cluster spec: featureSet: TechPreviewNoUpgrade
oc get machineconfignodes
NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED ci-ln-mdb23yt-72292-kzdsg-master-0 master rendered-master-f21b093d20f68a7c06f922ed3ea5fbc8 rendered-master-1abc053eec29e6c945670f39d6dc8afa False ci-ln-mdb23yt-72292-kzdsg-master-1 master rendered-master-1abc053eec29e6c945670f39d6dc8afa rendered-master-1abc053eec29e6c945670f39d6dc8afa True ci-ln-mdb23yt-72292-kzdsg-master-2 master rendered-master-1abc053eec29e6c945670f39d6dc8afa rendered-master-1abc053eec29e6c945670f39d6dc8afa True ci-ln-mdb23yt-72292-kzdsg-worker-a-gfqjr worker rendered-worker-d0130cd74e9e576d7ba78ce166272bfb rendered-worker-8f61bf839898a4487c3b5263a430e94a False ci-ln-mdb23yt-72292-kzdsg-worker-b-gknq4 worker rendered-worker-8f61bf839898a4487c3b5263a430e94a rendered-worker-8f61bf839898a4487c3b5263a430e94a True ci-ln-mdb23yt-72292-kzdsg-worker-c-mffrx worker rendered-worker-8f61bf839898a4487c3b5263a430e94a rendered-worker-8f61bf839898a4487c3b5263a430e94a True
oc get machineconfignodes -o wide
NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED UPDATEPREPARED UPDATEEXECUTED UPDATEPOSTACTIONCOMPLETE UPDATECOMPLETE RESUMED UPDATECOMPATIBLE UPDATEDFILESANDOS CORDONEDNODE DRAINEDNODE REBOOTEDNODE RELOADEDCRIO UNCORDONEDNODE ci-ln-g6dr34b-72292-g9btv-master-0 master rendered-master-d4e122320b351cdbe1df59ddb63ddcfc rendered-master-6f2064fcb36d2a914de5b0c660dc49ff False True Unknown False False False True Unknown False False False False False ci-ln-g6dr34b-72292-g9btv-master-1 master rendered-master-6f2064fcb36d2a914de5b0c660dc49ff rendered-master-6f2064fcb36d2a914de5b0c660dc49ff True False False False False False False False False False False False False ci-ln-g6dr34b-72292-g9btv-master-2 master rendered-master-6f2064fcb36d2a914de5b0c660dc49ff rendered-master-6f2064fcb36d2a914de5b0c660dc49ff True False False False False False False False False False False False False ci-ln-g6dr34b-72292-g9btv-worker-a-sjh5r worker rendered-worker-671b88c8c569fa3f60dc1a27cf9c91f2 rendered-worker-d5534cb730e5e108905fc285c2a42b6c False True Unknown False False False True Unknown False False False False False ci-ln-g6dr34b-72292-g9btv-worker-b-xthbz worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-d5534cb730e5e108905fc285c2a42b6c True False False False False False False False False False False False False ci-ln-g6dr34b-72292-g9btv-worker-c-gnpd6 worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-d5534cb730e5e108905fc285c2a42b6c True False False False False False False False False False False False False
oc get machineconfignodes USD(oc get machineconfignodes -o json | jq -r '.items[]|select(.spec.pool.name=="<pool_name>")|.metadata.name') 1
NAME POOLNAME DESIREDCONFIG CURRENTCONFIG UPDATED ci-ln-g6dr34b-72292-g9btv-worker-a-sjh5r worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-d5534cb730e5e108905fc285c2a42b6c True ci-ln-g6dr34b-72292-g9btv-worker-b-xthbz worker rendered-worker-d5534cb730e5e108905fc285c2a42b6c rendered-worker-faf6b50218a8bbce21f1370866283de5 False ci-ln-g6dr34b-72292-g9btv-worker-c-gnpd6 worker rendered-worker-faf6b50218a8bbce21f1370866283de5 rendered-worker-faf6b50218a8bbce21f1370866283de5 True
oc describe machineconfignode/<node_name>
Name: <node_name> Namespace: Labels: <none> Annotations: <none> API Version: machineconfiguration.openshift.io/v1alpha1 Kind: MachineConfigNode Metadata: Creation Timestamp: 2023-10-17T13:08:58Z Generation: 1 Resource Version: 49443 UID: 4bd758ab-2187-413c-ac42-882e61761b1d Spec: Node Ref: Name: <node_name> Pool: Name: master ConfigVersion: Desired: rendered-worker-823ff8dc2b33bf444709ed7cd2b9855b Status: Message: Drained node. The drain is complete as the desired drainer matches current drainer: drain-rendered-worker-01f27f752eb84eba917450e43636b210 Reason: UpdateExecutedDrained Status: True Type: Drained Last Transition Time: 2025-01-14T15:45:55Z Config Version: Current: rendered-master-8110974a5cea69dff5b263237b58abd8 Desired: rendered-worker-823ff8dc2b33bf444709ed7cd2b9855b Observed Generation: 6
oc get controllerconfig/machine-config-controller -o yaml | yq -y '.status.controllerCertificates'
- bundleFile: KubeAPIServerServingCAData notAfter: '2034-10-23T13:13:02Z' notBefore: '2024-10-25T13:13:02Z' signer: CN=admin-kubeconfig-signer,OU=openshift subject: CN=admin-kubeconfig-signer,OU=openshift - bundleFile: KubeAPIServerServingCAData notAfter: '2024-10-26T13:13:05Z' notBefore: '2024-10-25T13:27:14Z' signer: CN=kubelet-signer,OU=openshift subject: CN=kube-csr-signer_@1729862835 - bundleFile: KubeAPIServerServingCAData notAfter: '2024-10-26T13:13:05Z' notBefore: '2024-10-25T13:13:05Z' signer: CN=kubelet-signer,OU=openshift subject: CN=kubelet-signer,OU=openshift
oc get mcp master -o yaml | yq -y '.status.certExpirys'
- bundle: KubeAPIServerServingCAData expiry: '2034-10-23T13:13:02Z' subject: CN=admin-kubeconfig-signer,OU=openshift - bundle: KubeAPIServerServingCAData expiry: '2024-10-26T13:13:05Z' subject: CN=kube-csr-signer_@1729862835 - bundle: KubeAPIServerServingCAData expiry: '2024-10-26T13:13:05Z' subject: CN=kubelet-signer,OU=openshift - bundle: KubeAPIServerServingCAData expiry: '2025-10-25T13:13:05Z' subject: CN=kube-apiserver-to-kubelet-signer,OU=openshift
oc debug node/<node_name>
sh-5.1# chroot /host
sh-5.1# ls /etc/docker/certs.d
image-registry.openshift-image-registry.svc.cluster.local:5000 image-registry.openshift-image-registry.svc:5000