Chapter 10. Changing the cloud provider credentials configuration For supported configurations, you can change how OpenShift Container Platform authenticates with your cloud provider. To determine which cloud credentials strategy your cluster uses, see Determining the Cloud Credential Operator mode . 10.1. Rotating cloud provider service keys with the Cloud Credential Operator utility Some organizations require the rotation of the service keys that authenticate the cluster. You can use the Cloud Credential Operator (CCO) utility ( ccoctl ) to update keys for clusters installed on the following cloud providers: Amazon Web Services (AWS) with Security Token Service (STS) Google Cloud Platform (GCP) with GCP Workload Identity Microsoft Azure with Workload ID IBM Cloud 10.1.1. Rotating AWS OIDC bound service account signer keys If the Cloud Credential Operator (CCO) for your OpenShift Container Platform cluster on Amazon Web Services (AWS) is configured to operate in manual mode with STS, you can rotate the bound service account signer key. To rotate the key, you delete the existing key on your cluster, which causes the Kubernetes API server to create a new key. To reduce authentication failures during this process, you must immediately add the new public key to the existing issuer file. After the cluster is using the new key for authentication, you can remove any remaining keys. Important The process to rotate OIDC bound service account signer keys is disruptive and takes a significant amount of time. Some steps are time-sensitive. Before proceeding, observe the following considerations: Read the following steps and ensure that you understand and accept the time requirement. The exact time requirement varies depending on the individual cluster, but it is likely to require at least one hour. To reduce the risk of authentication failures, ensure that you understand and prepare for the time-sensitive steps. During this process, you must refresh all service accounts and restart all pods on the cluster. These actions are disruptive to workloads. To mitigate this impact, you can temporarily halt these services and then redeploy them when the cluster is ready. Prerequisites You have access to the OpenShift CLI ( oc ) as a user with the cluster-admin role. You have created an AWS account for the ccoctl utility to use with the following permissions: s3:GetObject s3:PutObject s3:PutObjectTagging For clusters that store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL, the AWS account that runs the ccoctl utility requires the cloudfront:ListDistributions permission. You have configured the ccoctl utility. Your cluster is in a stable state. You can confirm that the cluster is stable by running the following command: USD oc adm wait-for-stable-cluster --minimum-stable-period=5s Procedure Configure the following environment variables: INFRA_ID=USD(oc get infrastructures cluster -o jsonpath='{.status.infrastructureName}') CLUSTER_NAME=USD{INFRA_ID%-*} 1 1 1 This value should match the name of the cluster that was specified in the metadata.name field of the install-config.yaml file during installation. Note Your cluster might differ from this example, and the resource names might not be derived identically from the cluster name. Ensure that you specify the correct corresponding resource names for your cluster. For AWS clusters that store the OIDC configuration in a public S3 bucket, configure the following environment variable: AWS_BUCKET=USD(oc get authentication cluster -o jsonpath={'.spec.serviceAccountIssuer'} | awk -F'://' '{printUSD2}' |awk -F'.' '{printUSD1}') For AWS clusters that store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL, complete the following steps: Extract the public CloudFront distribution URL by running the following command: USD basename USD(oc get authentication cluster -o jsonpath={'.spec.serviceAccountIssuer'} ) Example output <subdomain>.cloudfront.net where <subdomain> is an alphanumeric string. Determine the private S3 bucket name by running the following command: USD aws cloudfront list-distributions --query "DistributionList.Items[].{DomainName: DomainName, OriginDomainName: Origins.Items[0].DomainName}[?contains(DomainName, '<subdomain>.cloudfront.net')]" Example output [ { "DomainName": "<subdomain>.cloudfront.net", "OriginDomainName": "<s3_bucket>.s3.us-east-2.amazonaws.com" } ] where <s3_bucket> is the private S3 bucket name for your cluster. Configure the following environment variable: AWS_BUCKET=USD<s3_bucket> where <s3_bucket> is the private S3 bucket name for your cluster. Create a temporary directory to use and assign it an environment variable by running the following command: USD TEMPDIR=USD(mktemp -d) To cause the Kubernetes API server to create a new bound service account signing key, you delete the bound service account signing key. Important After you complete this step, the Kubernetes API server starts to roll out a new key. To reduce the risk of authentication failures, complete the remaining steps as quickly as possible. The remaining steps might be disruptive to workloads. When you are ready, delete the bound service account signing key by running the following command: USD oc delete secrets/-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator Download the public key from the service account signing key secret that the Kubernetes API server created by running the following command: USD oc get secret/-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 \ -d > USD{TEMPDIR}/serviceaccount-signer.public Use the public key to create a keys.json file by running the following command: USD ccoctl aws create-identity-provider \ --dry-run \ 1 --output-dir USD{TEMPDIR} \ --name fake \ 2 --region us-east-1 3 1 The --dry-run option outputs files, including the new keys.json file, to the disk without making API calls. 2 Because the --dry-run option does not make any API calls, some parameters do not require real values. 3 Specify any valid AWS region, such as us-east-1 . This value does not need to match the region the cluster is in. Rename the keys.json file by running the following command: USD cp USD{TEMPDIR}/<number>-keys.json USD{TEMPDIR}/jwks.new.json where <number> is a two-digit numerical value that varies depending on your environment. Download the existing keys.json file from the cloud provider by running the following command: USD aws s3api get-object \ --bucket USD{AWS_BUCKET} \ --key keys.json USD{TEMPDIR}/jwks.current.json Combine the two keys.json files by running the following command: USD jq -s '{ keys: map(.keys[])}' USD{TEMPDIR}/jwks.current.json USD{TEMPDIR}/jwks.new.json > USD{TEMPDIR}/jwks.combined.json To enable authentication for the old and new keys during the rotation, upload the combined keys.json file to the cloud provider by running the following command: USD aws s3api put-object \ --bucket USD{AWS_BUCKET} \ --tagging "openshift.io/cloud-credential-operator/USD{CLUSTER_NAME}=owned" \ --key keys.json \ --body USD{TEMPDIR}/jwks.combined.json Wait for the Kubernetes API server to update and use the new key. You can monitor the update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable To ensure that all pods on the cluster use the new key, you must restart them. Important This step maintains uptime for services that are configured for high availability across multiple nodes, but might cause downtime for any services that are not. Restart all of the pods in the cluster by running the following command: USD oc adm reboot-machine-config-pool mcp/worker mcp/master Monitor the restart and update process by running the following command: USD oc adm wait-for-node-reboot nodes --all This process might take 15 minutes or longer. The following output indicates that the process is complete: All nodes rebooted Monitor the update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable Replace the combined keys.json file with the updated keys.json file on the cloud provider by running the following command: USD aws s3api put-object \ --bucket USD{AWS_BUCKET} \ --tagging "openshift.io/cloud-credential-operator/USD{CLUSTER_NAME}=owned" \ --key keys.json \ --body USD{TEMPDIR}/jwks.new.json 10.1.2. Rotating GCP OIDC bound service account signer keys If the Cloud Credential Operator (CCO) for your OpenShift Container Platform cluster on Google Cloud Platform (GCP) is configured to operate in manual mode with GCP Workload Identity, you can rotate the bound service account signer key. To rotate the key, you delete the existing key on your cluster, which causes the Kubernetes API server to create a new key. To reduce authentication failures during this process, you must immediately add the new public key to the existing issuer file. After the cluster is using the new key for authentication, you can remove any remaining keys. Important The process to rotate OIDC bound service account signer keys is disruptive and takes a significant amount of time. Some steps are time-sensitive. Before proceeding, observe the following considerations: Read the following steps and ensure that you understand and accept the time requirement. The exact time requirement varies depending on the individual cluster, but it is likely to require at least one hour. To reduce the risk of authentication failures, ensure that you understand and prepare for the time-sensitive steps. During this process, you must refresh all service accounts and restart all pods on the cluster. These actions are disruptive to workloads. To mitigate this impact, you can temporarily halt these services and then redeploy them when the cluster is ready. Prerequisites You have access to the OpenShift CLI ( oc ) as a user with the cluster-admin role. You have added one of the following authentication options to the GCP account that the ccoctl utility uses: The IAM Workload Identity Pool Admin role The following granular permissions: storage.objects.create storage.objects.delete You have configured the ccoctl utility. Your cluster is in a stable state. You can confirm that the cluster is stable by running the following command: USD oc adm wait-for-stable-cluster --minimum-stable-period=5s Procedure Configure the following environment variables: CURRENT_ISSUER=USD(oc get authentication cluster -o jsonpath='{.spec.serviceAccountIssuer}') GCP_BUCKET=USD(echo USD{CURRENT_ISSUER} | cut -d "/" -f4) Note Your cluster might differ from this example, and the resource names might not be derived identically from the cluster name. Ensure that you specify the correct corresponding resource names for your cluster. Create a temporary directory to use and assign it an environment variable by running the following command: USD TEMPDIR=USD(mktemp -d) To cause the Kubernetes API server to create a new bound service account signing key, you delete the bound service account signing key. Important After you complete this step, the Kubernetes API server starts to roll out a new key. To reduce the risk of authentication failures, complete the remaining steps as quickly as possible. The remaining steps might be disruptive to workloads. When you are ready, delete the bound service account signing key by running the following command: USD oc delete secrets/-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator Download the public key from the service account signing key secret that the Kubernetes API server created by running the following command: USD oc get secret/-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 \ -d > USD{TEMPDIR}/serviceaccount-signer.public Use the public key to create a keys.json file by running the following command: USD ccoctl gcp create-workload-identity-provider \ --dry-run \ 1 --output-dir=USD{TEMPDIR} \ --name fake \ 2 --project fake \ --workload-identity-pool fake 1 The --dry-run option outputs files, including the new keys.json file, to the disk without making API calls. 2 Because the --dry-run option does not make any API calls, some parameters do not require real values. Rename the keys.json file by running the following command: USD cp USD{TEMPDIR}/<number>-keys.json USD{TEMPDIR}/jwks.new.json where <number> is a two-digit numerical value that varies depending on your environment. Download the existing keys.json file from the cloud provider by running the following command: USD gcloud storage cp gs://USD{GCP_BUCKET}/keys.json USD{TEMPDIR}/jwks.current.json Combine the two keys.json files by running the following command: USD jq -s '{ keys: map(.keys[])}' USD{TEMPDIR}/jwks.current.json USD{TEMPDIR}/jwks.new.json > USD{TEMPDIR}/jwks.combined.json To enable authentication for the old and new keys during the rotation, upload the combined keys.json file to the cloud provider by running the following command: USD gcloud storage cp USD{TEMPDIR}/jwks.combined.json gs://USD{GCP_BUCKET}/keys.json Wait for the Kubernetes API server to update and use the new key. You can monitor the update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable To ensure that all pods on the cluster use the new key, you must restart them. Important This step maintains uptime for services that are configured for high availability across multiple nodes, but might cause downtime for any services that are not. Restart all of the pods in the cluster by running the following command: USD oc adm reboot-machine-config-pool mcp/worker mcp/master Monitor the restart and update process by running the following command: USD oc adm wait-for-node-reboot nodes --all This process might take 15 minutes or longer. The following output indicates that the process is complete: All nodes rebooted Monitor the update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable Replace the combined keys.json file with the updated keys.json file on the cloud provider by running the following command: USD gcloud storage cp USD{TEMPDIR}/jwks.new.json gs://USD{GCP_BUCKET}/keys.json 10.1.3. Rotating Azure OIDC bound service account signer keys If the Cloud Credential Operator (CCO) for your OpenShift Container Platform cluster on Microsoft Azure is configured to operate in manual mode with Microsoft Entra Workload ID, you can rotate the bound service account signer key. To rotate the key, you delete the existing key on your cluster, which causes the Kubernetes API server to create a new key. To reduce authentication failures during this process, you must immediately add the new public key to the existing issuer file. After the cluster is using the new key for authentication, you can remove any remaining keys. Important The process to rotate OIDC bound service account signer keys is disruptive and takes a significant amount of time. Some steps are time-sensitive. Before proceeding, observe the following considerations: Read the following steps and ensure that you understand and accept the time requirement. The exact time requirement varies depending on the individual cluster, but it is likely to require at least one hour. To reduce the risk of authentication failures, ensure that you understand and prepare for the time-sensitive steps. During this process, you must refresh all service accounts and restart all pods on the cluster. These actions are disruptive to workloads. To mitigate this impact, you can temporarily halt these services and then redeploy them when the cluster is ready. Prerequisites You have access to the OpenShift CLI ( oc ) as a user with the cluster-admin role. You have created a global Azure account for the ccoctl utility to use with the following permissions: Microsoft.Storage/storageAccounts/listkeys/action Microsoft.Storage/storageAccounts/read Microsoft.Storage/storageAccounts/write Microsoft.Storage/storageAccounts/blobServices/containers/read Microsoft.Storage/storageAccounts/blobServices/containers/write You have configured the ccoctl utility. Your cluster is in a stable state. You can confirm that the cluster is stable by running the following command: USD oc adm wait-for-stable-cluster --minimum-stable-period=5s Procedure Configure the following environment variables: CURRENT_ISSUER=USD(oc get authentication cluster -o jsonpath='{.spec.serviceAccountIssuer}') AZURE_STORAGE_ACCOUNT=USD(echo USD{CURRENT_ISSUER} | cut -d "/" -f3 | cut -d "." -f1) AZURE_STORAGE_CONTAINER=USD(echo USD{CURRENT_ISSUER} | cut -d "/" -f4) Note Your cluster might differ from this example, and the resource names might not be derived identically from the cluster name. Ensure that you specify the correct corresponding resource names for your cluster. Create a temporary directory to use and assign it an environment variable by running the following command: USD TEMPDIR=USD(mktemp -d) To cause the Kubernetes API server to create a new bound service account signing key, you delete the bound service account signing key. Important After you complete this step, the Kubernetes API server starts to roll out a new key. To reduce the risk of authentication failures, complete the remaining steps as quickly as possible. The remaining steps might be disruptive to workloads. When you are ready, delete the bound service account signing key by running the following command: USD oc delete secrets/-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator Download the public key from the service account signing key secret that the Kubernetes API server created by running the following command: USD oc get secret/-bound-service-account-signing-key \ -n openshift-kube-apiserver-operator \ -ojsonpath='{ .data.service-account\.pub }' | base64 \ -d > USD{TEMPDIR}/serviceaccount-signer.public Use the public key to create a keys.json file by running the following command: USD ccoctl aws create-identity-provider \ 1 --dry-run \ 2 --output-dir USD{TEMPDIR} \ --name fake \ 3 --region us-east-1 4 1 The ccoctl azure command does not include a --dry-run option. To use the --dry-run option, you must specify aws for an Azure cluster. 2 The --dry-run option outputs files, including the new keys.json file, to the disk without making API calls. 3 Because the --dry-run option does not make any API calls, some parameters do not require real values. 4 Specify any valid AWS region, such as us-east-1 . This value does not need to match the region the cluster is in. Rename the keys.json file by running the following command: USD cp USD{TEMPDIR}/<number>-keys.json USD{TEMPDIR}/jwks.new.json where <number> is a two-digit numerical value that varies depending on your environment. Download the existing keys.json file from the cloud provider by running the following command: USD az storage blob download \ --container-name USD{AZURE_STORAGE_CONTAINER} \ --account-name USD{AZURE_STORAGE_ACCOUNT} \ --name 'openid/v1/jwks' \ -f USD{TEMPDIR}/jwks.current.json Combine the two keys.json files by running the following command: USD jq -s '{ keys: map(.keys[])}' USD{TEMPDIR}/jwks.current.json USD{TEMPDIR}/jwks.new.json > USD{TEMPDIR}/jwks.combined.json To enable authentication for the old and new keys during the rotation, upload the combined keys.json file to the cloud provider by running the following command: USD az storage blob upload \ --overwrite \ --account-name USD{AZURE_STORAGE_ACCOUNT} \ --container-name USD{AZURE_STORAGE_CONTAINER} \ --name 'openid/v1/jwks' \ -f USD{TEMPDIR}/jwks.combined.json Wait for the Kubernetes API server to update and use the new key. You can monitor the update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable To ensure that all pods on the cluster use the new key, you must restart them. Important This step maintains uptime for services that are configured for high availability across multiple nodes, but might cause downtime for any services that are not. Restart all of the pods in the cluster by running the following command: USD oc adm reboot-machine-config-pool mcp/worker mcp/master Monitor the restart and update process by running the following command: USD oc adm wait-for-node-reboot nodes --all This process might take 15 minutes or longer. The following output indicates that the process is complete: All nodes rebooted Monitor the update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable Replace the combined keys.json file with the updated keys.json file on the cloud provider by running the following command: USD az storage blob upload \ --overwrite \ --account-name USD{AZURE_STORAGE_ACCOUNT} \ --container-name USD{AZURE_STORAGE_CONTAINER} \ --name 'openid/v1/jwks' \ -f USD{TEMPDIR}/jwks.new.json 10.1.4. Rotating IBM Cloud credentials You can rotate API keys for your existing service IDs and update the corresponding secrets. Prerequisites You have configured the ccoctl utility. You have existing service IDs in a live OpenShift Container Platform cluster installed. Procedure Use the ccoctl utility to rotate your API keys for the service IDs and update the secrets by running the following command: USD ccoctl <provider_name> refresh-keys \ 1 --kubeconfig <openshift_kubeconfig_file> \ 2 --credentials-requests-dir <path_to_credential_requests_directory> \ 3 --name <name> 4 1 The name of the provider. For example: ibmcloud or powervs . 2 The kubeconfig file associated with the cluster. For example, <installation_directory>/auth/kubeconfig . 3 The directory where the credential requests are stored. 4 The name of the OpenShift Container Platform cluster. Note If your cluster uses Technology Preview features that are enabled by the TechPreviewNoUpgrade feature set, you must include the --enable-tech-preview parameter. 10.2. Rotating cloud provider credentials Some organizations require the rotation of the cloud provider credentials. To allow the cluster to use the new credentials, you must update the secrets that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials. 10.2.1. Rotating cloud provider credentials manually If your cloud provider credentials are changed for any reason, you must manually update the secret that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials. The process for rotating cloud credentials depends on the mode that the CCO is configured to use. After you rotate credentials for a cluster that is using mint mode, you must manually remove the component credentials that were created by the removed credential. Prerequisites Your cluster is installed on a platform that supports rotating cloud credentials manually with the CCO mode that you are using: For mint mode, Amazon Web Services (AWS) and Google Cloud Platform (GCP) are supported. For passthrough mode, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), Red Hat OpenStack Platform (RHOSP), and VMware vSphere are supported. You have changed the credentials that are used to interface with your cloud provider. The new credentials have sufficient permissions for the mode CCO is configured to use in your cluster. Procedure In the Administrator perspective of the web console, navigate to Workloads Secrets . In the table on the Secrets page, find the root secret for your cloud provider. Platform Secret name AWS aws-creds Azure azure-credentials GCP gcp-credentials RHOSP openstack-credentials VMware vSphere vsphere-creds Click the Options menu in the same row as the secret and select Edit Secret . Record the contents of the Value field or fields. You can use this information to verify that the value is different after updating the credentials. Update the text in the Value field or fields with the new authentication information for your cloud provider, and then click Save . If you are updating the credentials for a vSphere cluster that does not have the vSphere CSI Driver Operator enabled, you must force a rollout of the Kubernetes controller manager to apply the updated credentials. Note If the vSphere CSI Driver Operator is enabled, this step is not required. To apply the updated vSphere credentials, log in to the OpenShift Container Platform CLI as a user with the cluster-admin role and run the following command: USD oc patch kubecontrollermanager cluster \ -p='{"spec": {"forceRedeploymentReason": "recovery-'"USD( date )"'"}}' \ --type=merge While the credentials are rolling out, the status of the Kubernetes Controller Manager Operator reports Progressing=true . To view the status, run the following command: USD oc get co kube-controller-manager If the CCO for your cluster is configured to use mint mode, delete each component secret that is referenced by the individual CredentialsRequest objects. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role. Get the names and namespaces of all referenced component secrets: USD oc -n openshift-cloud-credential-operator get CredentialsRequest \ -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef' where <provider_spec> is the corresponding value for your cloud provider: AWS: AWSProviderSpec GCP: GCPProviderSpec Partial example output for AWS { "name": "ebs-cloud-credentials", "namespace": "openshift-cluster-csi-drivers" } { "name": "cloud-credential-operator-iam-ro-creds", "namespace": "openshift-cloud-credential-operator" } Delete each of the referenced component secrets: USD oc delete secret <secret_name> \ 1 -n <secret_namespace> 2 1 Specify the name of a secret. 2 Specify the namespace that contains the secret. Example deletion of an AWS secret USD oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers You do not need to manually delete the credentials from your provider console. Deleting the referenced component secrets will cause the CCO to delete the existing credentials from the platform and create new ones. Verification To verify that the credentials have changed: In the Administrator perspective of the web console, navigate to Workloads Secrets . Verify that the contents of the Value field or fields have changed. Additional resources The Cloud Credential Operator in mint mode The Cloud Credential Operator in passthrough mode vSphere CSI Driver Operator 10.3. Removing cloud provider credentials After installing OpenShift Container Platform, some organizations require the removal of the cloud provider credentials that were used during the initial installation. To allow the cluster to use the new credentials, you must update the secrets that the Cloud Credential Operator (CCO) uses to manage cloud provider credentials. 10.3.1. Removing cloud provider credentials For clusters that use the Cloud Credential Operator (CCO) in mint mode, the administrator-level credential is stored in the kube-system namespace. The CCO uses the admin credential to process the CredentialsRequest objects in the cluster and create users for components with limited permissions. After installing an OpenShift Container Platform cluster with the CCO in mint mode, you can remove the administrator-level credential secret from the kube-system namespace in the cluster. The CCO only requires the administrator-level credential during changes that require reconciling new or modified CredentialsRequest custom resources, such as minor cluster version updates. Note Before performing a minor version cluster update (for example, updating from OpenShift Container Platform 4.17 to 4.18), you must reinstate the credential secret with the administrator-level credential. If the credential is not present, the update might be blocked. Prerequisites Your cluster is installed on a platform that supports removing cloud credentials from the CCO. Supported platforms are AWS and GCP. Procedure In the Administrator perspective of the web console, navigate to Workloads Secrets . In the table on the Secrets page, find the root secret for your cloud provider. Platform Secret name AWS aws-creds GCP gcp-credentials Click the Options menu in the same row as the secret and select Delete Secret . Additional resources The Cloud Credential Operator in mint mode 10.4. Enabling token-based authentication After installing an Microsoft Azure OpenShift Container Platform cluster, you can enable Microsoft Entra Workload ID to use short-term credentials. 10.4.1. Configuring the Cloud Credential Operator utility To configure an existing cluster to create and manage cloud credentials from outside of the cluster, extract and prepare the Cloud Credential Operator utility ( ccoctl ) binary. Note The ccoctl utility is a Linux binary that must run in a Linux environment. Prerequisites You have access to an OpenShift Container Platform account with cluster administrator access. You have installed the OpenShift CLI ( oc ). Procedure Set a variable for the OpenShift Container Platform release image by running the following command: USD RELEASE_IMAGE=USD(oc get clusterversion -o jsonpath={..desired.image}) Obtain the CCO container image from the OpenShift Container Platform release image by running the following command: USD CCO_IMAGE=USD(oc adm release info --image-for='cloud-credential-operator' USDRELEASE_IMAGE -a ~/.pull-secret) Note Ensure that the architecture of the USDRELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool. Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command: USD oc image extract USDCCO_IMAGE \ --file="/usr/bin/ccoctl.<rhel_version>" \ 1 -a ~/.pull-secret 1 For <rhel_version> , specify the value that corresponds to the version of Red Hat Enterprise Linux (RHEL) that the host uses. If no value is specified, ccoctl.rhel8 is used by default. The following values are valid: rhel8 : Specify this value for hosts that use RHEL 8. rhel9 : Specify this value for hosts that use RHEL 9. Change the permissions to make ccoctl executable by running the following command: USD chmod 775 ccoctl.<rhel_version> Verification To verify that ccoctl is ready to use, display the help file. Use a relative file name when you run the command, for example: USD ./ccoctl.rhel9 Example output OpenShift credentials provisioning tool Usage: ccoctl [command] Available Commands: aws Manage credentials objects for AWS cloud azure Manage credentials objects for Azure gcp Manage credentials objects for Google cloud help Help about any command ibmcloud Manage credentials objects for {ibm-cloud-title} nutanix Manage credentials objects for Nutanix Flags: -h, --help help for ccoctl Use "ccoctl [command] --help" for more information about a command. 10.4.2. Enabling Microsoft Entra Workload ID on an existing cluster If you did not configure your Microsoft Azure OpenShift Container Platform cluster to use Microsoft Entra Workload ID during installation, you can enable this authentication method on an existing cluster. Important The process to enable Workload ID on an existing cluster is disruptive and takes a significant amount of time. Before proceeding, observe the following considerations: Read the following steps and ensure that you understand and accept the time requirement. The exact time requirement varies depending on the individual cluster, but it is likely to require at least one hour. During this process, you must refresh all service accounts and restart all pods on the cluster. These actions are disruptive to workloads. To mitigate this impact, you can temporarily halt these services and then redeploy them when the cluster is ready. After starting this process, do not attempt to update the cluster until it is complete. If an update is triggered, the process to enable Workload ID on an existing cluster fails. Prerequisites You have installed an OpenShift Container Platform cluster on Microsoft Azure. You have access to the cluster using an account with cluster-admin permissions. You have installed the OpenShift CLI ( oc ). You have extracted and prepared the Cloud Credential Operator utility ( ccoctl ) binary. You have access to your Azure account by using the Azure CLI ( az ). Procedure Create an output directory for the manifests that the ccoctl utility generates. This procedure uses ./output_dir as an example. Extract the service account public signing key for the cluster to the output directory by running the following command: USD oc get configmap \ --namespace openshift-kube-apiserver bound-sa-token-signing-certs \ --output 'go-template={{index .data "service-account-001.pub"}}' > ./output_dir/serviceaccount-signer.public 1 1 This procedure uses a file named serviceaccount-signer.public as an example. Use the extracted service account public signing key to create an OpenID Connect (OIDC) issuer and Azure blob storage container with OIDC configuration files by running the following command: USD ./ccoctl azure create-oidc-issuer \ --name <azure_infra_name> \ 1 --output-dir ./output_dir \ --region <azure_region> \ 2 --subscription-id <azure_subscription_id> \ 3 --tenant-id <azure_tenant_id> \ --public-key-file ./output_dir/serviceaccount-signer.public 4 1 The value of the name parameter is used to create an Azure resource group. To use an existing Azure resource group instead of creating a new one, specify the --oidc-resource-group-name argument with the existing group name as its value. 2 Specify the region of the existing cluster. 3 Specify the subscription ID of the existing cluster. 4 Specify the file that contains the service account public signing key for the cluster. Verify that the configuration file for the Azure pod identity webhook was created by running the following command: USD ll ./output_dir/manifests Example output total 8 -rw-------. 1 cloud-user cloud-user 193 May 22 02:29 azure-ad-pod-identity-webhook-config.yaml 1 -rw-------. 1 cloud-user cloud-user 165 May 22 02:29 cluster-authentication-02-config.yaml 1 The file azure-ad-pod-identity-webhook-config.yaml contains the Azure pod identity webhook configuration. Set an OIDC_ISSUER_URL variable with the OIDC issuer URL from the generated manifests in the output directory by running the following command: USD OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print USD2 }' ./output_dir/manifests/cluster-authentication-02-config.yaml` Update the spec.serviceAccountIssuer parameter of the cluster authentication configuration by running the following command: USD oc patch authentication cluster \ --type=merge \ -p "{\"spec\":{\"serviceAccountIssuer\":\"USD{OIDC_ISSUER_URL}\"}}" Monitor the configuration update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable Restart all of the pods in the cluster by running the following command: USD oc adm reboot-machine-config-pool mcp/worker mcp/master Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key. Monitor the restart and update process by running the following command: USD oc adm wait-for-node-reboot nodes --all This process might take 15 minutes or longer. The following output indicates that the process is complete: All nodes rebooted Update the Cloud Credential Operator spec.credentialsMode parameter to Manual by running the following command: USD oc patch cloudcredential cluster \ --type=merge \ --patch '{"spec":{"credentialsMode":"Manual"}}' Extract the list of CredentialsRequest objects from the OpenShift Container Platform release image by running the following command: USD oc adm release extract \ --credentials-requests \ --included \ --to <path_to_directory_for_credentials_requests> \ --registry-config ~/.pull-secret Note This command might take a few moments to run. Set an AZURE_INSTALL_RG variable with the Azure resource group name by running the following command: USD AZURE_INSTALL_RG=`oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'` Use the ccoctl utility to create managed identities for all CredentialsRequest objects by running the following command: USD ccoctl azure create-managed-identities \ --name <azure_infra_name> \ --output-dir ./output_dir \ --region <azure_region> \ --subscription-id <azure_subscription_id> \ --credentials-requests-dir <path_to_directory_for_credentials_requests> \ --issuer-url "USD{OIDC_ISSUER_URL}" \ --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \ 1 --installation-resource-group-name "USD{AZURE_INSTALL_RG}" 1 Specify the name of the resource group that contains the DNS zone. Apply the Azure pod identity webhook configuration for Workload ID by running the following command: USD oc apply -f ./output_dir/manifests/azure-ad-pod-identity-webhook-config.yaml Apply the secrets generated by the ccoctl utility by running the following command: USD find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {} This process might take several minutes. Restart all of the pods in the cluster by running the following command: USD oc adm reboot-machine-config-pool mcp/worker mcp/master Restarting a pod updates the serviceAccountIssuer field and refreshes the service account public signing key. Monitor the restart and update process by running the following command: USD oc adm wait-for-node-reboot nodes --all This process might take 15 minutes or longer. The following output indicates that the process is complete: All nodes rebooted Monitor the configuration update progress by running the following command: USD oc adm wait-for-stable-cluster This process might take 15 minutes or longer. The following output indicates that the process is complete: All clusteroperators are stable Optional: Remove the Azure root credentials secret by running the following command: USD oc delete secret -n kube-system azure-credentials Additional resources Microsoft Entra Workload ID Configuring an Azure cluster to use short-term credentials 10.4.3. Verifying that a cluster uses short-term credentials You can verify that a cluster uses short-term security credentials for individual components by checking the Cloud Credential Operator (CCO) configuration and other values in the cluster. Prerequisites You deployed an OpenShift Container Platform cluster using the Cloud Credential Operator utility ( ccoctl ) to implement short-term credentials. You installed the OpenShift CLI ( oc ). You are logged in as a user with cluster-admin privileges. Procedure Verify that the CCO is configured to operate in manual mode by running the following command: USD oc get cloudcredentials cluster \ -o=jsonpath={.spec.credentialsMode} The following output confirms that the CCO is operating in manual mode: Example output Manual Verify that the cluster does not have root credentials by running the following command: USD oc get secrets \ -n kube-system <secret_name> where <secret_name> is the name of the root secret for your cloud provider. Platform Secret name Amazon Web Services (AWS) aws-creds Microsoft Azure azure-credentials Google Cloud Platform (GCP) gcp-credentials An error confirms that the root secret is not present on the cluster. Example output for an AWS cluster Error from server (NotFound): secrets "aws-creds" not found Verify that the components are using short-term security credentials for individual components by running the following command: USD oc get authentication cluster \ -o jsonpath \ --template='{ .spec.serviceAccountIssuer }' This command displays the value of the .spec.serviceAccountIssuer parameter in the cluster Authentication object. An output of a URL that is associated with your cloud provider indicates that the cluster is using manual mode with short-term credentials that are created and managed from outside of the cluster. Azure clusters: Verify that the components are assuming the Azure client ID that is specified in the secret manifests by running the following command: USD oc get secrets \ -n openshift-image-registry installer-cloud-credentials \ -o jsonpath='{.data}' An output that contains the azure_client_id and azure_federated_token_file felids confirms that the components are assuming the Azure client ID. Azure clusters: Verify that the pod identity webhook is running by running the following command: USD oc get pods \ -n openshift-cloud-credential-operator Example output NAME READY STATUS RESTARTS AGE cloud-credential-operator-59cf744f78-r8pbq 2/2 Running 2 71m pod-identity-webhook-548f977b4c-859lz 1/1 Running 1 70m 10.5. Additional resources About the Cloud Credential Operator

Commands:
oc adm wait-for-stable-cluster --minimum-stable-period=5s
INFRA_ID=USD(oc get infrastructures cluster -o jsonpath='{.status.infrastructureName}') CLUSTER_NAME=USD{INFRA_ID%-*} 1
AWS_BUCKET=USD(oc get authentication cluster -o jsonpath={'.spec.serviceAccountIssuer'} | awk -F'://' '{printUSD2}' |awk -F'.' '{printUSD1}')
basename USD(oc get authentication cluster -o jsonpath={'.spec.serviceAccountIssuer'} )
<subdomain>.cloudfront.net
aws cloudfront list-distributions --query "DistributionList.Items[].{DomainName: DomainName, OriginDomainName: Origins.Items[0].DomainName}[?contains(DomainName, '<subdomain>.cloudfront.net')]"
DomainName: <subdomain>.cloudfront.net
OriginDomainName: <s3_bucket>.s3.us-east-2.amazonaws.com
AWS_BUCKET=USD<s3_bucket>
TEMPDIR=USD(mktemp -d)
oc delete secrets/next-bound-service-account-signing-key -n openshift-kube-apiserver-operator
oc get secret/next-bound-service-account-signing-key -n openshift-kube-apiserver-operator -ojsonpath='{ .data.service-account\.pub }' | base64 -d > USD{TEMPDIR}/serviceaccount-signer.public
ccoctl aws create-identity-provider --dry-run \ 1 --output-dir USD{TEMPDIR} --name fake \ 2 --region us-east-1 3
cp USD{TEMPDIR}/<number>-keys.json USD{TEMPDIR}/jwks.new.json
aws s3api get-object --bucket USD{AWS_BUCKET} --key keys.json USD{TEMPDIR}/jwks.current.json
jq -s '{ keys: map(.keys[])}' USD{TEMPDIR}/jwks.current.json USD{TEMPDIR}/jwks.new.json > USD{TEMPDIR}/jwks.combined.json
aws s3api put-object --bucket USD{AWS_BUCKET} --tagging "openshift.io/cloud-credential-operator/USD{CLUSTER_NAME}=owned" --key keys.json --body USD{TEMPDIR}/jwks.combined.json
oc adm wait-for-stable-cluster
All clusteroperators are stable
oc adm reboot-machine-config-pool mcp/worker mcp/master
oc adm wait-for-node-reboot nodes --all
All nodes rebooted
oc adm wait-for-stable-cluster
All clusteroperators are stable
aws s3api put-object --bucket USD{AWS_BUCKET} --tagging "openshift.io/cloud-credential-operator/USD{CLUSTER_NAME}=owned" --key keys.json --body USD{TEMPDIR}/jwks.new.json
oc adm wait-for-stable-cluster --minimum-stable-period=5s
CURRENT_ISSUER=USD(oc get authentication cluster -o jsonpath='{.spec.serviceAccountIssuer}') GCP_BUCKET=USD(echo USD{CURRENT_ISSUER} | cut -d "/" -f4)
TEMPDIR=USD(mktemp -d)
oc delete secrets/next-bound-service-account-signing-key -n openshift-kube-apiserver-operator
oc get secret/next-bound-service-account-signing-key -n openshift-kube-apiserver-operator -ojsonpath='{ .data.service-account\.pub }' | base64 -d > USD{TEMPDIR}/serviceaccount-signer.public
ccoctl gcp create-workload-identity-provider --dry-run \ 1 --output-dir=USD{TEMPDIR} --name fake \ 2 --project fake --workload-identity-pool fake
cp USD{TEMPDIR}/<number>-keys.json USD{TEMPDIR}/jwks.new.json
gcloud storage cp gs://USD{GCP_BUCKET}/keys.json USD{TEMPDIR}/jwks.current.json
jq -s '{ keys: map(.keys[])}' USD{TEMPDIR}/jwks.current.json USD{TEMPDIR}/jwks.new.json > USD{TEMPDIR}/jwks.combined.json
gcloud storage cp USD{TEMPDIR}/jwks.combined.json gs://USD{GCP_BUCKET}/keys.json
oc adm wait-for-stable-cluster
All clusteroperators are stable
oc adm reboot-machine-config-pool mcp/worker mcp/master
oc adm wait-for-node-reboot nodes --all
All nodes rebooted
oc adm wait-for-stable-cluster
All clusteroperators are stable
gcloud storage cp USD{TEMPDIR}/jwks.new.json gs://USD{GCP_BUCKET}/keys.json
oc adm wait-for-stable-cluster --minimum-stable-period=5s
CURRENT_ISSUER=USD(oc get authentication cluster -o jsonpath='{.spec.serviceAccountIssuer}') AZURE_STORAGE_ACCOUNT=USD(echo USD{CURRENT_ISSUER} | cut -d "/" -f3 | cut -d "." -f1) AZURE_STORAGE_CONTAINER=USD(echo USD{CURRENT_ISSUER} | cut -d "/" -f4)
TEMPDIR=USD(mktemp -d)
oc delete secrets/next-bound-service-account-signing-key -n openshift-kube-apiserver-operator
oc get secret/next-bound-service-account-signing-key -n openshift-kube-apiserver-operator -ojsonpath='{ .data.service-account\.pub }' | base64 -d > USD{TEMPDIR}/serviceaccount-signer.public
ccoctl aws create-identity-provider \ 1 --dry-run \ 2 --output-dir USD{TEMPDIR} --name fake \ 3 --region us-east-1 4
cp USD{TEMPDIR}/<number>-keys.json USD{TEMPDIR}/jwks.new.json
az storage blob download --container-name USD{AZURE_STORAGE_CONTAINER} --account-name USD{AZURE_STORAGE_ACCOUNT} --name 'openid/v1/jwks' -f USD{TEMPDIR}/jwks.current.json
jq -s '{ keys: map(.keys[])}' USD{TEMPDIR}/jwks.current.json USD{TEMPDIR}/jwks.new.json > USD{TEMPDIR}/jwks.combined.json
az storage blob upload --overwrite --account-name USD{AZURE_STORAGE_ACCOUNT} --container-name USD{AZURE_STORAGE_CONTAINER} --name 'openid/v1/jwks' -f USD{TEMPDIR}/jwks.combined.json
oc adm wait-for-stable-cluster
All clusteroperators are stable
oc adm reboot-machine-config-pool mcp/worker mcp/master
oc adm wait-for-node-reboot nodes --all
All nodes rebooted
oc adm wait-for-stable-cluster
All clusteroperators are stable
az storage blob upload --overwrite --account-name USD{AZURE_STORAGE_ACCOUNT} --container-name USD{AZURE_STORAGE_CONTAINER} --name 'openid/v1/jwks' -f USD{TEMPDIR}/jwks.new.json
ccoctl <provider_name> refresh-keys \ 1 --kubeconfig <openshift_kubeconfig_file> \ 2 --credentials-requests-dir <path_to_credential_requests_directory> \ 3 --name <name> 4
oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"USD( date )"'"}}' --type=merge
oc get co kube-controller-manager
oc -n openshift-cloud-credential-operator get CredentialsRequest -o json | jq -r '.items[] | select (.spec.providerSpec.kind=="<provider_spec>") | .spec.secretRef'
{ "name": "ebs-cloud-credentials", "namespace": "openshift-cluster-csi-drivers" } { "name": "cloud-credential-operator-iam-ro-creds", "namespace": "openshift-cloud-credential-operator" }
oc delete secret <secret_name> \ 1 -n <secret_namespace> 2
oc delete secret ebs-cloud-credentials -n openshift-cluster-csi-drivers
RELEASE_IMAGE=USD(oc get clusterversion -o jsonpath={..desired.image})
CCO_IMAGE=USD(oc adm release info --image-for='cloud-credential-operator' USDRELEASE_IMAGE -a ~/.pull-secret)
oc image extract USDCCO_IMAGE --file="/usr/bin/ccoctl.<rhel_version>" \ 1 -a ~/.pull-secret
chmod 775 ccoctl.<rhel_version>
./ccoctl.rhel9
OpenShift credentials provisioning tool Usage: ccoctl [command] Available Commands: aws Manage credentials objects for AWS cloud azure Manage credentials objects for Azure gcp Manage credentials objects for Google cloud help Help about any command ibmcloud Manage credentials objects for {ibm-cloud-title} nutanix Manage credentials objects for Nutanix Flags: -h, --help help for ccoctl Use "ccoctl [command] --help" for more information about a command.
oc get configmap --namespace openshift-kube-apiserver bound-sa-token-signing-certs --output 'go-template={{index .data "service-account-001.pub"}}' > ./output_dir/serviceaccount-signer.public 1
./ccoctl azure create-oidc-issuer --name <azure_infra_name> \ 1 --output-dir ./output_dir --region <azure_region> \ 2 --subscription-id <azure_subscription_id> \ 3 --tenant-id <azure_tenant_id> --public-key-file ./output_dir/serviceaccount-signer.public 4
ll ./output_dir/manifests
total 8 -rw-------. 1 cloud-user cloud-user 193 May 22 02:29 azure-ad-pod-identity-webhook-config.yaml 1 -rw-------. 1 cloud-user cloud-user 165 May 22 02:29 cluster-authentication-02-config.yaml
OIDC_ISSUER_URL=`awk '/serviceAccountIssuer/ { print USD2 }' ./output_dir/manifests/cluster-authentication-02-config.yaml`
oc patch authentication cluster --type=merge -p "{\"spec\":{\"serviceAccountIssuer\":\"USD{OIDC_ISSUER_URL}\"}}"
oc adm wait-for-stable-cluster
All clusteroperators are stable
oc adm reboot-machine-config-pool mcp/worker mcp/master
oc adm wait-for-node-reboot nodes --all
All nodes rebooted
oc patch cloudcredential cluster --type=merge --patch '{"spec":{"credentialsMode":"Manual"}}'
oc adm release extract --credentials-requests --included --to <path_to_directory_for_credentials_requests> --registry-config ~/.pull-secret
AZURE_INSTALL_RG=`oc get infrastructure cluster -o jsonpath --template '{ .status.platformStatus.azure.resourceGroupName }'`
ccoctl azure create-managed-identities --name <azure_infra_name> --output-dir ./output_dir --region <azure_region> --subscription-id <azure_subscription_id> --credentials-requests-dir <path_to_directory_for_credentials_requests> --issuer-url "USD{OIDC_ISSUER_URL}" --dnszone-resource-group-name <azure_dns_zone_resourcegroup_name> \ 1 --installation-resource-group-name "USD{AZURE_INSTALL_RG}"
oc apply -f ./output_dir/manifests/azure-ad-pod-identity-webhook-config.yaml
find ./output_dir/manifests -iname "openshift*yaml" -print0 | xargs -I {} -0 -t oc replace -f {}
oc adm reboot-machine-config-pool mcp/worker mcp/master
oc adm wait-for-node-reboot nodes --all
All nodes rebooted
oc adm wait-for-stable-cluster
All clusteroperators are stable
oc delete secret -n kube-system azure-credentials
oc get cloudcredentials cluster -o=jsonpath={.spec.credentialsMode}
Manual
oc get secrets -n kube-system <secret_name>
Error from server (NotFound): secrets "aws-creds" not found
oc get authentication cluster -o jsonpath --template='{ .spec.serviceAccountIssuer }'
oc get secrets -n openshift-image-registry installer-cloud-credentials -o jsonpath='{.data}'
oc get pods -n openshift-cloud-credential-operator
NAME READY STATUS RESTARTS AGE cloud-credential-operator-59cf744f78-r8pbq 2/2 Running 2 71m pod-identity-webhook-548f977b4c-859lz 1/1 Running 1 70m