Updating clusters OpenShift Container Platform 4.18 Updating OpenShift Container Platform clusters Red Hat OpenShift Documentation Team

Commands:
oc adm upgrade --include-not-recommended
Cluster version is 4.13.40 Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.14 (available channels: candidate-4.13, candidate-4.14, eus-4.14, fast-4.13, fast-4.14, stable-4.13, stable-4.14) Recommended updates: VERSION IMAGE 4.14.27 quay.io/openshift-release-dev/ocp-release@sha256:4d30b359aa6600a89ed49ce6a9a5fdab54092bcb821a25480fdfbc47e66af9ec 4.14.26 quay.io/openshift-release-dev/ocp-release@sha256:4fe7d4ccf4d967a309f83118f1a380a656a733d7fcee1dbaf4d51752a6372890 4.14.25 quay.io/openshift-release-dev/ocp-release@sha256:a0ef946ef8ae75aef726af1d9bbaad278559ad8cab2c1ed1088928a0087990b6 4.14.24 quay.io/openshift-release-dev/ocp-release@sha256:0a34eac4b834e67f1bca94493c237e307be2c0eae7b8956d4d8ef1c0c462c7b0 4.14.23 quay.io/openshift-release-dev/ocp-release@sha256:f8465817382128ec7c0bc676174bad0fb43204c353e49c146ddd83a5b3d58d92 4.13.42 quay.io/openshift-release-dev/ocp-release@sha256:dcf5c3ad7384f8bee3c275da8f886b0bc9aea7611d166d695d0cf0fff40a0b55 4.13.41 quay.io/openshift-release-dev/ocp-release@sha256:dbb8aa0cf53dc5ac663514e259ad2768d8c82fd1fe7181a4cfb484e3ffdbd3ba Updates with known issues: Version: 4.14.22 Image: quay.io/openshift-release-dev/ocp-release@sha256:7093fa606debe63820671cc92a1384e14d0b70058d4b4719d666571e1fc62190 Reason: MultipleReasons Message: Exposure to AzureRegistryImageMigrationUserProvisioned is unknown due to an evaluation failure: client-side throttling: only 18.061ms has elapsed since the last match call completed for this cluster condition backend; this cached cluster condition request has been queued for later execution In Azure clusters with the user-provisioned registry storage, the in-cluster image registry component may struggle to complete the cluster update. https://issues.redhat.com/browse/IR-468 Incoming HTTP requests to services exposed by Routes may fail while routers reload their configuration, especially when made with Apache HTTPClient versions before 5.0. The problem is more likely to occur in clusters with higher number of Routes and corresponding endpoints. https://issues.redhat.com/browse/NE-1689 Version: 4.14.21 Image: quay.io/openshift-release-dev/ocp-release@sha256:6e3fba19a1453e61f8846c6b0ad3abf41436a3550092cbfd364ad4ce194582b7 Reason: MultipleReasons Message: Exposure to AzureRegistryImageMigrationUserProvisioned is unknown due to an evaluation failure: client-side throttling: only 33.991ms has elapsed since the last match call completed for this cluster condition backend; this cached cluster condition request has been queued for later execution In Azure clusters with the user-provisioned registry storage, the in-cluster image registry component may struggle to complete the cluster update. https://issues.redhat.com/browse/IR-468 Incoming HTTP requests to services exposed by Routes may fail while routers reload their configuration, especially when made with Apache HTTPClient versions before 5.0. The problem is more likely to occur in clusters with higher number of Routes and corresponding endpoints. https://issues.redhat.com/browse/NE-1689
oc get clusterversion version -o json | jq '.status.availableUpdates'
[ { "channels": [ "candidate-4.11", "candidate-4.12", "fast-4.11", "fast-4.12" ], "image": "quay.io/openshift-release-dev/ocp-release@sha256:400267c7f4e61c6bfa0a59571467e8bd85c9188e442cbd820cc8263809be3775", "url": "https://access.redhat.com/errata/RHBA-2023:3213", "version": "4.11.41" }, ]
oc get clusterversion version -o json | jq '.status.conditionalUpdates'
[ { "conditions": [ { "lastTransitionTime": "2023-05-30T16:28:59Z", "message": "The 4.11.36 release only resolves an installation issue https://issues.redhat.com//browse/OCPBUGS-11663 , which does not affect already running clusters. 4.11.36 does not include fixes delivered in recent 4.11.z releases and therefore upgrading from these versions would cause fixed bugs to reappear. Red Hat does not recommend upgrading clusters to 4.11.36 version for this reason. https://access.redhat.com/solutions/7007136", "reason": "PatchesOlderRelease", "status": "False", "type": "Recommended" } ], "release": { "channels": [...], "image": "quay.io/openshift-release-dev/ocp-release@sha256:8c04176b771a62abd801fcda3e952633566c8b5ff177b93592e8e8d2d1f8471d", "url": "https://access.redhat.com/errata/RHBA-2023:1733", "version": "4.11.36" }, "risks": [...] }, ]
oc adm release extract <release image>
oc adm release extract quay.io/openshift-release-dev/ocp-release:4.12.6-x86_64 Extracted release payload from digest sha256:800d1e39d145664975a3bb7cbc6e674fbf78e3c45b5dde9ff2c5a11a8690c87b created at 2023-03-01T12:46:29Z ls 0000_03_authorization-openshift_01_rolebindingrestriction.crd.yaml 0000_03_config-operator_01_proxy.crd.yaml 0000_03_marketplace-operator_01_operatorhub.crd.yaml 0000_03_marketplace-operator_02_operatorhub.cr.yaml 0000_03_quota-openshift_01_clusterresourcequota.crd.yaml 1 0000_90_service-ca-operator_02_prometheusrolebinding.yaml 2 0000_90_service-ca-operator_03_servicemonitor.yaml 0000_99_machine-api-operator_00_tombstones.yaml image-references 3 release-metadata
0000_<runlevel>_<component>_<manifest-name>.yaml
0000_03_config-operator_01_proxy.crd.yaml
oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-acd1358917e9f98cbdb599aea622d78b True False False 3 3 3 0 22h worker rendered-worker-1d871ac76e1951d32b2fe92369879826 False True False 2 1 1 0 22h
oc adm upgrade channel <channel>
oc get node
NAME STATUS ROLES AGE VERSION ip-10-0-137-31.us-east-2.compute.internal Ready,SchedulingDisabled worker 12d v1.23.5+3afdacb ip-10-0-151-208.us-east-2.compute.internal Ready master 12d v1.23.5+3afdacb ip-10-0-176-138.us-east-2.compute.internal Ready master 12d v1.23.5+3afdacb ip-10-0-183-194.us-east-2.compute.internal Ready worker 12d v1.23.5+3afdacb ip-10-0-204-102.us-east-2.compute.internal Ready master 12d v1.23.5+3afdacb ip-10-0-207-224.us-east-2.compute.internal Ready worker 12d v1.23.5+3afdacb
Cluster update time = CVO target update payload deployment time + (# node update iterations x MCO node update time)
Cluster update time = 60 + (6 x 5) = 90 minutes
Cluster update time = 60 + (3 x 5) = 75 minutes
oc get cloudcredentials cluster -o=jsonpath={.spec.credentialsMode}
oc get secret <secret_name> -n=kube-system
oc get authentication cluster -o jsonpath --template='{ .spec.serviceAccountIssuer }'
oc adm upgrade
Recommended updates: VERSION IMAGE 4.18.0 quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032
RELEASE_IMAGE=<update_pull_spec>
quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032
oc adm release extract --from=USDRELEASE_IMAGE --credentials-requests --included \ 1 --to=<path_to_directory_for_credentials_requests> 2
apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: cloud-credential-operator-iam-ro namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - iam:GetUser - iam:GetUserPolicy - iam:ListAccessKeys resource: "*" secretRef: name: cloud-credential-operator-iam-ro-creds namespace: openshift-cloud-credential-operator 1
oc create namespace <component_namespace>
RELEASE_IMAGE=USD(oc get clusterversion -o jsonpath={..desired.image})
CCO_IMAGE=USD(oc adm release info --image-for='cloud-credential-operator' USDRELEASE_IMAGE -a ~/.pull-secret)
oc image extract USDCCO_IMAGE --file="/usr/bin/ccoctl.<rhel_version>" \ 1 -a ~/.pull-secret
chmod 775 ccoctl.<rhel_version>
./ccoctl.rhel9
OpenShift credentials provisioning tool Usage: ccoctl [command] Available Commands: aws Manage credentials objects for AWS cloud azure Manage credentials objects for Azure gcp Manage credentials objects for Google cloud help Help about any command ibmcloud Manage credentials objects for {ibm-cloud-title} nutanix Manage credentials objects for Nutanix Flags: -h, --help help for ccoctl Use "ccoctl [command] --help" for more information about a command.
ccoctl aws create-all \ 1 --name=<name> \ 2 --region=<aws_region> \ 3 --credentials-requests-dir=<path_to_credentials_requests_directory> \ 4 --output-dir=<path_to_ccoctl_output_dir> \ 5 --create-private-s3-bucket 6
ccoctl gcp create-all --name=<name> \ 1 --region=<gcp_region> \ 2 --project=<gcp_project_id> \ 3 --credentials-requests-dir=<path_to_credentials_requests_directory> \ 4 --output-dir=<path_to_ccoctl_output_dir> 5
ccoctl ibmcloud create-service-id --credentials-requests-dir=<path_to_credential_requests_directory> \ 1 --name=<cluster_name> \ 2 --output-dir=<installation_directory> \ 3 --resource-group-name=<resource_group_name> 4
ccoctl azure create-managed-identities --name <azure_infra_name> \ 1 --output-dir ./output_dir --region <azure_region> \ 2 --subscription-id <azure_subscription_id> \ 3 --credentials-requests-dir <path_to_directory_for_credentials_requests> --issuer-url "USD{OIDC_ISSUER_URL}" \ 4 --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \ 5 --installation-resource-group-name "USD{AZURE_INSTALL_RG}" 6
oc get authentication cluster -o jsonpath --template='{ .spec.serviceAccountIssuer }'
oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'
ccoctl nutanix create-shared-secrets --credentials-requests-dir=<path_to_credentials_requests_directory> \ 1 --output-dir=<ccoctl_output_dir> \ 2 --credentials-source-filepath=<path_to_credentials_file> 3
ls <path_to_ccoctl_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}
apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: <component_credentials_request> namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AWSProviderSpec statementEntries: - effect: Allow action: - s3:CreateBucket - s3:DeleteBucket resource: "*" secretRef: name: <component_secret> namespace: <component_namespace>
apiVersion: v1 kind: Secret metadata: name: <component_secret> namespace: <component_namespace> data: aws_access_key_id: <base64_encoded_aws_access_key_id> aws_secret_access_key: <base64_encoded_aws_secret_access_key>
apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: <component_credentials_request> namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: AzureProviderSpec roleBindings: - role: Contributor secretRef: name: <component_secret> namespace: <component_namespace>
apiVersion: v1 kind: Secret metadata: name: <component_secret> namespace: <component_namespace> data: azure_subscription_id: <base64_encoded_azure_subscription_id> azure_client_id: <base64_encoded_azure_client_id> azure_client_secret: <base64_encoded_azure_client_secret> azure_tenant_id: <base64_encoded_azure_tenant_id> azure_resource_prefix: <base64_encoded_azure_resource_prefix> azure_resourcegroup: <base64_encoded_azure_resourcegroup> azure_region: <base64_encoded_azure_region>
apiVersion: cloudcredential.openshift.io/v1 kind: CredentialsRequest metadata: name: <component_credentials_request> namespace: openshift-cloud-credential-operator spec: providerSpec: apiVersion: cloudcredential.openshift.io/v1 kind: GCPProviderSpec predefinedRoles: - roles/iam.securityReviewer - roles/iam.roleViewer skipServiceCheck: true secretRef: name: <component_secret> namespace: <component_namespace>
apiVersion: v1 kind: Secret metadata: name: <component_secret> namespace: <component_namespace> data: service_account.json: <base64_encoded_gcp_service_account_file>
oc edit cloudcredential cluster
metadata: annotations: cloudcredential.openshift.io/upgradeable-to: <version_number>
RUN depmod -b /opt USD{KERNEL_VERSION}
quay.io/openshift-release-dev/ocp-release@sha256:22e149142517dfccb47be828f012659b1ccf71d26620e6f62468c264a7ce7863
apiVersion: kmm.sigs.x-k8s.io/v1beta2 kind: PreflightValidationOCP metadata: name: preflight spec: releaseImage: quay.io/openshift-release-dev/ocp-release@sha256:22e149142517dfccb47be828f012659b1ccf71d26620e6f62468c264a7ce7863 pushBuiltImage: true
oc get machinehealthcheck -n openshift-machine-api
oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused=""
apiVersion: machine.openshift.io/v1beta1 kind: MachineHealthCheck metadata: name: example namespace: openshift-machine-api annotations: cluster.x-k8s.io/paused: "" spec: selector: matchLabels: role: worker unhealthyConditions: - type: "Ready" status: "Unknown" timeout: "300s" - type: "Ready" status: "False" timeout: "300s" maxUnhealthy: "40%" status: currentHealthy: 5 expectedMachines: 5
oc -n openshift-machine-api annotate mhc <mhc-name> cluster.x-k8s.io/paused-
oc adm upgrade
Cluster version is 4.13.10 Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.13 (available channels: candidate-4.13, candidate-4.14, fast-4.13, stable-4.13) Recommended updates: VERSION IMAGE 4.13.14 quay.io/openshift-release-dev/ocp-release@sha256:406fcc160c097f61080412afcfa7fd65284ac8741ac7ad5b480e304aba73674b 4.13.13 quay.io/openshift-release-dev/ocp-release@sha256:d62495768e335c79a215ba56771ff5ae97e3cbb2bf49ed8fb3f6cefabcdc0f17 4.13.12 quay.io/openshift-release-dev/ocp-release@sha256:73946971c03b43a0dc6f7b0946b26a177c2f3c9d37105441315b4e3359373a55 4.13.11 quay.io/openshift-release-dev/ocp-release@sha256:e1c2377fdae1d063aaddc753b99acf25972b6997ab9a0b7e80cfef627b9ef3dd
oc adm upgrade channel <channel>
oc adm upgrade channel stable-4.18
oc adm upgrade --to-latest=true 1
oc adm upgrade --to=<version> 1
oc adm upgrade
oc adm upgrade
Cluster version is <version> Upstream is unset, so the cluster will use an appropriate default. Channel: stable-<version> (available channels: candidate-<version>, eus-<version>, fast-<version>, stable-<version>) No updates available. You may force an update to a specific release image, but doing so might not be supported and might result in downtime or data loss.
oc get nodes
NAME STATUS ROLES AGE VERSION ip-10-0-168-251.ec2.internal Ready master 82m v1.31.3 ip-10-0-170-223.ec2.internal Ready master 82m v1.31.3 ip-10-0-179-95.ec2.internal Ready worker 70m v1.31.3 ip-10-0-182-134.ec2.internal Ready worker 70m v1.31.3 ip-10-0-211-16.ec2.internal Ready master 82m v1.31.3 ip-10-0-250-100.ec2.internal Ready worker 69m v1.31.3
export OC_ENABLE_CMD_UPGRADE_RECOMMEND=true
oc adm upgrade recommend
Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.13 (available channels: candidate-4.12, candidate-4.13, eus-4.12, eus-4.14, fast-4.12, fast-4.13, stable-4.12, stable-4.13) Updates to 4.13: Version: 4.13.50 Image: quay.io/openshift-release-dev/ocp-release@sha256:6afb11e1cac46fd26476ca134072937115256b9c6360f7a1cd1812992c065f02 Reason: AdminAckRequired Message: Kubernetes 1.26 and therefore OpenShift 4.13 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6958394 for details and instructions. Version: 4.13.49 Image: quay.io/openshift-release-dev/ocp-release@sha256:ab6f574665395d809511db9dc57764358278538eaae248c6d199208b3c30ab7d Reason: AdminAckRequired Message: Kubernetes 1.26 and therefore OpenShift 4.13 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6958394 for details and instructions. Updates to 4.12: VERSION ISSUES 4.12.64 no known issues 4.12.63 no known issues And 43 older 4.12 updates you can see with '--show-outdated-releases' or '--version VERSION'.
oc adm upgrade recommend --version 4.12.51
Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.13 (available channels: candidate-4.12, candidate-4.13, eus-4.12, eus-4.14, fast-4.12, fast-4.13, stable-4.12, stable-4.13) Update to 4.12.51 Recommended=False: Image: quay.io/openshift-release-dev/ocp-release@sha256:158ced797e49f6caf7862acccef58484be63b642fdd2f66e6416295fa7958ab0 Release URL: https://access.redhat.com/errata/RHSA-2024:1052 Reason: MultipleReasons Message: An unintended reversion to the default kubelet nodeStatusReportFrequency can cause significant load on the control plane. https://issues.redhat.com/browse/MCO-1094 After rebooting into kernel-4.18.0-372.88.1.el8_6 or later, kernel nodes experience high load average and io_wait times. The nodes might fail to start or stop pods and probes may fail. Workload and host processes may become unresponsive and workload may be disrupted. https://issues.redhat.com/browse/COS-2705
export OC_ENABLE_CMD_UPGRADE_STATUS=true
oc adm upgrade status
= Control Plane = Assessment: Progressing Target Version: 4.17.1 (from 4.17.0) Updating: machine-config Completion: 97% (32 operators updated, 1 updating, 0 waiting) Duration: 54m (Est. Time Remaining: <10m) Operator Status: 32 Healthy, 1 Unavailable Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-53-40.us-east-2.compute.internal Progressing Draining 4.17.0 +10m ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.17.0 ? ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.17.0 ? = Worker Upgrade = WORKER POOL ASSESSMENT COMPLETION STATUS worker Progressing 0% (0/2) 1 Available, 1 Progressing, 1 Draining infra Progressing 50% (1/2) 1 Available, 1 Progressing, 1 Draining Worker Pool Nodes: Worker NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-4-159.us-east-2.compute.internal Progressing Draining 4.17.0 +10m ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.17.0 ? Worker Pool Nodes: infra NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-4-159-infra.us-east-2.compute.internal Progressing Draining 4.17.0 +10m ip-10-0-20-162.us-east-2.compute.internal Completed Updated 4.17.1 - = Update Health = SINCE LEVEL IMPACT MESSAGE 54m4s Info None Update is proceeding well
oc adm upgrade --include-not-recommended
oc adm upgrade --allow-not-recommended --to <version> <.>
oc patch clusterversion/version --patch '{"spec":{"upstream":"<update-server-url>"}}' --type=merge
clusterversion.config.openshift.io/version patched
spec: clusterID: db93436d-7b05-42cc-b856-43e11ad2d31a upstream: '<update-server-url>' 1
oc get mcp
NAME CONFIG UPDATED UPDATING master rendered-master-ecbb9582781c1091e1c9f19d50cf836c True False worker rendered-worker-00a3f0c68ae94e747193156b491553d5 True False
oc adm upgrade channel eus-<4.y+2>
oc patch mcp/worker --type merge --patch '{"spec":{"paused":true}}'
oc adm upgrade --to-latest
Updating to latest version <4.y+1.z>
oc adm upgrade
Cluster version is <4.y+1.z>
oc adm upgrade --to-latest
oc adm upgrade
Cluster version is <4.y+2.z>
oc patch mcp/worker --type merge --patch '{"spec":{"paused":false}}'
oc get mcp
NAME CONFIG UPDATED UPDATING master rendered-master-52da4d2760807cb2b96a3402179a9a4c True False worker rendered-worker-4756f60eccae96fb9dcb4c392c69d497 True False
oc get -l 'node-role.kubernetes.io/master!=' -o 'jsonpath={range .items[*]}{.metadata.name}{"\n"}{end}' nodes
ci-ln-pwnll6b-f76d1-s8t9n-worker-a-s75z4 ci-ln-pwnll6b-f76d1-s8t9n-worker-b-dglj2 ci-ln-pwnll6b-f76d1-s8t9n-worker-c-lldbm
oc label node <node_name> node-role.kubernetes.io/<custom_label>=
oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary=
node/ci-ln-gtrwm8t-f76d1-spbl7-worker-a-xk76k labeled
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: name: workerpool-canary 1 spec: machineConfigSelector: matchExpressions: - { key: machineconfiguration.openshift.io/role, operator: In, values: [worker,workerpool-canary] 2 } nodeSelector: matchLabels: node-role.kubernetes.io/workerpool-canary: "" 3
oc create -f <file_name>
machineconfigpool.machineconfiguration.openshift.io/workerpool-canary created
oc get machineconfigpool
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-b0bb90c4921860f2a5d8a2f8137c1867 True False False 3 3 3 0 97m workerpool-canary rendered-workerpool-canary-87ba3dec1ad78cb6aecebf7fbb476a36 True False False 1 1 1 0 2m42s worker rendered-worker-87ba3dec1ad78cb6aecebf7fbb476a36 True False False 2 2 2 0 97m
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: name: worker-perf spec: machineConfigSelector: matchExpressions: - { key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-perf] } nodeSelector: matchLabels: node-role.kubernetes.io/worker-perf: ""
oc create -f machineConfigPool.yaml
machineconfigpool.machineconfiguration.openshift.io/worker-perf created
oc label node worker-a node-role.kubernetes.io/worker-perf=''
oc label node worker-b node-role.kubernetes.io/worker-perf=''
oc label node worker-c node-role.kubernetes.io/worker-perf=''
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker-perf name: 06-kdump-enable-worker-perf spec: config: ignition: version: 3.2.0 systemd: units: - enabled: true name: kdump.service kernelArguments: - crashkernel=512M
oc create -f new-machineconfig.yaml
oc label node worker-a node-role.kubernetes.io/worker-perf-canary=''
oc label node worker-a node-role.kubernetes.io/worker-perf-
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: name: worker-perf-canary spec: machineConfigSelector: matchExpressions: - { key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-perf,worker-perf-canary] 1 } nodeSelector: matchLabels: node-role.kubernetes.io/worker-perf-canary: ""
oc create -f machineConfigPool-Canary.yaml
machineconfigpool.machineconfiguration.openshift.io/worker-perf-canary created
oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-2bf1379b39e22bae858ea1a3ff54b2ac True False False 3 3 3 0 5d16h worker rendered-worker-b9576d51e030413cfab12eb5b9841f34 True False False 0 0 0 0 5d16h worker-perf rendered-worker-perf-b98a1f62485fa702c4329d17d9364f6a True False False 2 2 2 0 56m worker-perf-canary rendered-worker-perf-canary-b98a1f62485fa702c4329d17d9364f6a True False False 1 1 1 0 44m
oc get nodes
NAME STATUS ROLES AGE VERSION worker-a Ready worker,worker-perf-canary 5d15h v1.27.13+e709aa5 worker-b Ready worker,worker-perf 5d15h v1.27.13+e709aa5 worker-c Ready worker,worker-perf 5d15h v1.27.13+e709aa5
systemctl status kdump.service
NAME STATUS ROLES AGE VERSION kdump.service - Crash recovery kernel arming Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; preset: disabled) Active: active (exited) since Tue 2024-09-03 12:44:43 UTC; 10s ago Process: 4151139 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS) Main PID: 4151139 (code=exited, status=0/SUCCESS)
cat /proc/cmdline
crashkernel=512M
oc label node worker-a node-role.kubernetes.io/worker-perf=''
oc label node worker-a node-role.kubernetes.io/worker-perf-canary-
oc patch mcp/<mcp_name> --patch '{"spec":{"paused":true}}' --type=merge
oc patch mcp/workerpool-canary --patch '{"spec":{"paused":true}}' --type=merge
machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched
oc patch mcp/<mcp_name> --patch '{"spec":{"paused":false}}' --type=merge
oc patch mcp/workerpool-canary --patch '{"spec":{"paused":false}}' --type=merge
machineconfigpool.machineconfiguration.openshift.io/workerpool-canary patched
oc get machineconfigpools
oc label node <node_name> node-role.kubernetes.io/<custom_label>-
oc label node ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz node-role.kubernetes.io/workerpool-canary-
node/ci-ln-0qv1yp2-f76d1-kl2tq-worker-a-j2ssz labeled
oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-1203f157d053fd987c7cbd91e3fbc0ed True False False 3 3 3 0 61m workerpool-canary rendered-mcp-noupdate-5ad4791166c468f3a35cd16e734c9028 True False False 0 0 0 0 21m worker rendered-worker-5ad4791166c468f3a35cd16e734c9028 True False False 3 3 3 0 61m
oc delete mcp <mcp_name>
--- Trivial example forcing an operator to acknowledge the start of an upgrade file=/home/user/openshift-ansible/hooks/pre_compute.yml - name: note the start of a compute machine update debug: msg: "Compute machine upgrade of {{ inventory_hostname }} is about to start" - name: require the user agree to start an upgrade pause: prompt: "Press Enter to start the compute machine update"
[all:vars] openshift_node_pre_upgrade_hook=/home/user/openshift-ansible/hooks/pre_node.yml openshift_node_post_upgrade_hook=/home/user/openshift-ansible/hooks/post_node.yml
systemctl disable --now firewalld.service
subscription-manager repos --disable=rhocp-4.17-for-rhel-8-x86_64-rpms --enable=rhocp-4.18-for-rhel-8-x86_64-rpms
yum swap ansible ansible-core
yum update openshift-ansible openshift-clients
subscription-manager repos --disable=rhocp-4.17-for-rhel-8-x86_64-rpms --enable=rhocp-4.18-for-rhel-8-x86_64-rpms
[all:vars] ansible_user=root #ansible_become=True openshift_kubeconfig_path="~/.kube/config" [workers] mycluster-rhel8-0.example.com mycluster-rhel8-1.example.com mycluster-rhel8-2.example.com mycluster-rhel8-3.example.com
cd /usr/share/ansible/openshift-ansible
ansible-playbook -i /<path>/inventory/hosts playbooks/upgrade.yml 1
oc get node
NAME STATUS ROLES AGE VERSION mycluster-control-plane-0 Ready master 145m v1.31.3 mycluster-control-plane-1 Ready master 145m v1.31.3 mycluster-control-plane-2 Ready master 145m v1.31.3 mycluster-rhel8-0 Ready worker 98m v1.31.3 mycluster-rhel8-1 Ready worker 98m v1.31.3 mycluster-rhel8-2 Ready worker 98m v1.31.3 mycluster-rhel8-3 Ready worker 98m v1.31.3
yum update
oc get nodes -l node-role.kubernetes.io/master
NAME STATUS ROLES AGE VERSION control-plane-node-0 Ready master 75m v1.31.3 control-plane-node-1 Ready master 75m v1.31.3 control-plane-node-2 Ready master 75m v1.31.3
oc adm cordon <control_plane_node>
oc wait --for=condition=Ready node/<control_plane_node>
oc adm uncordon <control_plane_node>
oc get nodes -l node-role.kubernetes.io/worker
NAME STATUS ROLES AGE VERSION compute-node-0 Ready worker 30m v1.31.3 compute-node-1 Ready worker 30m v1.31.3 compute-node-2 Ready worker 30m v1.31.3
oc adm cordon <compute_node>
oc adm drain <compute_node> [--pod-selector=<pod_selector>]
oc wait --for=condition=Ready node/<compute_node>
oc adm uncordon <compute_node>
oc get clusterversion/version -o=jsonpath="{.status.conditions[?(.type=='RetrievedUpdates')].status}"
oc adm upgrade
oc adm upgrade channel <channel>
oc adm upgrade --to-multi-arch
oc adm upgrade
working towards USD{VERSION}: 106 of 841 done (12% complete), waiting on machine-config
oc adm upgrade status
oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME worker-001.example.com Ready worker 100d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x worker-002.example.com Ready worker 98d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x worker-003.example.com Ready worker 98d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x master-001.example.com Ready control-plane,master 120d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x master-002.example.com Ready control-plane,master 120d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x master-003.example.com Ready control-plane,master 120d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x
oc adm release info -o jsonpath="{ .metadata.metadata}"
release.openshift.io/architecture: multi
url: https://access.redhat.com/errata/<errata_version>
oc import-image <multiarch_image_stream_tag> --from=<registry>/<project_name>/<image_name> --import-mode='PreserveOriginal'
oc get configmap/coreos-bootimages -n openshift-machine-config-operator -o jsonpath='{.data.stream}' | jq -r '.architectures.aarch64.images.aws.regions."<aws_region>".image' 1
oc get infrastructure cluster -o jsonpath='{.status.platformStatus.aws.region}'
ami-xxxxxxx
oc edit controlplanemachineset.machine.openshift.io cluster -n openshift-machine-api
oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME worker-001.example.com Ready worker 100d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x worker-002.example.com Ready worker 98d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x worker-003.example.com Ready worker 98d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.x86_64 cri-o://1.30.x master-001.example.com Ready control-plane,master 120d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.aarch64 cri-o://1.30.x master-002.example.com Ready control-plane,master 120d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.aarch64 cri-o://1.30.x master-003.example.com Ready control-plane,master 120d v1.30.7 10.x.x.x <none> Red Hat Enterprise Linux CoreOS 4xx.xx.xxxxx-0 5.x.x-xxx.x.x.el9_xx.aarch64 cri-o://1.30.x
# bootupctl status
Component EFI Installed: grub2-efi-x64-1:2.04-31.el8_4.1.x86_64,shim-x64-15-8.el8_1.x86_64 Update: At latest version
Component EFI Installed: grub2-efi-aa64-1:2.02-99.el8_4.1.aarch64,shim-aa64-15.4-2.el8_1.aarch64 Update: At latest version
# bootupctl adopt-and-update
Updated: grub2-efi-x64-1:2.04-31.el8_4.1.x86_64,shim-x64-15-8.el8_1.x86_64
# bootupctl update
Updated: grub2-efi-x64-1:2.04-31.el8_4.1.x86_64,shim-x64-15-8.el8_1.x86_64
variant: openshift version: 4.18.0 metadata: name: 99-worker-chrony 1 labels: machineconfiguration.openshift.io/role: worker 2 systemd: units: - name: bootupctl-update.service enabled: true contents: | [Unit] Description=Bootupd automatic update [Service] ExecStart=/usr/bin/bootupctl update RemainAfterExit=yes [Install] WantedBy=multi-user.target
butane 99-worker-bootupctl-update.bu -o 99-worker-bootupctl-update.yaml
oc apply -f ./99-worker-bootupctl-update.yaml
export OC_ENABLE_CMD_UPGRADE_STATUS=true
oc adm upgrade status
= Control Plane = Assessment: Progressing Target Version: 4.17.1 (from 4.17.0) Updating: machine-config Completion: 97% (32 operators updated, 1 updating, 0 waiting) Duration: 54m (Est. Time Remaining: <10m) Operator Status: 32 Healthy, 1 Unavailable Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-53-40.us-east-2.compute.internal Progressing Draining 4.17.0 +10m ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.17.0 ? ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.17.0 ? = Worker Upgrade = WORKER POOL ASSESSMENT COMPLETION STATUS worker Progressing 0% (0/2) 1 Available, 1 Progressing, 1 Draining infra Progressing 50% (1/2) 1 Available, 1 Progressing, 1 Draining Worker Pool Nodes: Worker NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-4-159.us-east-2.compute.internal Progressing Draining 4.17.0 +10m ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.17.0 ? Worker Pool Nodes: infra NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-4-159-infra.us-east-2.compute.internal Progressing Draining 4.17.0 +10m ip-10-0-20-162.us-east-2.compute.internal Completed Updated 4.17.1 - = Update Health = SINCE LEVEL IMPACT MESSAGE 54m4s Info None Update is proceeding well
oc describe clusterversions/version
Desired: Channels: candidate-4.13 candidate-4.14 fast-4.13 fast-4.14 stable-4.13 Image: quay.io/openshift-release-dev/ocp-release@sha256:a148b19231e4634196717c3597001b7d0af91bf3a887c03c444f59d9582864f4 URL: https://access.redhat.com/errata/RHSA-2023:6130 Version: 4.13.19 History: Completion Time: 2023-11-07T20:26:04Z Image: quay.io/openshift-release-dev/ocp-release@sha256:a148b19231e4634196717c3597001b7d0af91bf3a887c03c444f59d9582864f4 Started Time: 2023-11-07T19:11:36Z State: Completed Verified: true Version: 4.13.19 Completion Time: 2023-10-04T18:53:29Z Image: quay.io/openshift-release-dev/ocp-release@sha256:eac141144d2ecd6cf27d24efe9209358ba516da22becc5f0abc199d25a9cfcec Started Time: 2023-10-04T17:26:31Z State: Completed Verified: true Version: 4.13.13 Completion Time: 2023-09-26T14:21:43Z Image: quay.io/openshift-release-dev/ocp-release@sha256:371328736411972e9640a9b24a07be0af16880863e1c1ab8b013f9984b4ef727 Started Time: 2023-09-26T14:02:33Z State: Completed Verified: false Version: 4.13.12 Observed Generation: 4 Version Hash: CMLl3sLq-EA= Events: <none>