Chapter 1. Upgrading by using the Operator Upgrades through the Red Hat Advanced Cluster Security for Kubernetes (RHACS) Operator are performed automatically or manually, depending on the Update approval option you chose at installation. Follow these guidelines when upgrading: If the version for Central is earlier than 3.74, you must upgrade to 3.74 before upgrading to a 4.x version. For upgrading Central to version 3.74, see the upgrade documentation for version 3.74 . When upgrading Operator-based Central deployments from version 3.74, first ensure the Operator upgrade mode is set to Manual . Then, upgrade the Operator to version 4.0 following the procedure in the upgrade documentation for version 4.0 and ensure that Central is online. After the upgrade to version 4.0 is complete, Red Hat recommends upgrading Central to the latest version for full functionality. 1.1. Preparing to upgrade Before you upgrade the Red Hat Advanced Cluster Security for Kubernetes (RHACS) version, complete the following steps: If you are upgrading from version 3.74, verify that you are running the latest patch release version of the RHACS Operator 3.74. Backup your existing Central database. If the cluster you are upgrading contains the SecuredCluster custom resource (CR), change the collection method to CORE_BPF . For more information, see "Changing the collection method". 1.1.1. Changing the collection method If the cluster that you are upgrading contains the SecuredCluster CR, you must ensure that the per node collection setting is set to CORE_BPF before you upgrade. Procedure In the OpenShift Container Platform web console, go to the RHACS Operator page. In the top navigation menu, select Secured Cluster . Click the instance name, for example, stackrox-secured-cluster-services . Use one of the following methods to change the setting: In the Form view , under Per Node Settings Collector Settings Collection , select CORE_BPF . Click YAML to open the YAML editor and locate the spec.perNode.collector.collection attribute. If the value is KernelModule or EBPF , then change it to CORE_BPF . Click Save. Additional resources Updating installed Operators Backing up Red Hat Advanced Cluster Security for Kubernetes 1.2. Modifying Central custom resource The Central DB service requires persistent storage. If you have not configured a default storage class for the Central cluster that is an SSD or is high performance, you must update the Central custom resource to configure the storage class for the Central DB persistent volume claim (PVC). Note Skip this section if you have already configured a default storage class for Central. Procedure Update the central custom resource with the following configuration: 1 You must not change the value of IsEnabled to Enabled . 2 If this claim exists, your cluster uses the existing claim, otherwise it creates a new claim. 1.3. Modifying Central custom resource for external database Prerequisites You must have a database in your database instance that supports PostgreSQL 13 and a user with the following permissions: Connection rights to the database. Usage and Create on the schema. Select , Insert , Update , and Delete on all tables in the schema. Usage on all sequences in the schema. Procedure Create a password secret in the deployed namespace by using the OpenShift Container Platform web console or the terminal. On the OpenShift Container Platform web console, go to the Workloads Secrets page. Create a Key/Value secret with the key password and the value as the path of a plain text file containing the password for the superuser of the provisioned database. Or, run the following command in your terminal: USD oc create secret generic external-db-password \ 1 --from-file=password=<password.txt> 2 1 If you use Kubernetes, enter kubectl instead of oc . 2 Replace password.txt with the path of the file which has the plain text password. Go to the Red Hat Advanced Cluster Security for Kubernetes operator page in the OpenShift Container Platform web console. Select Central in the top navigation bar and select the instance you want to connect to the database. Go to the YAML editor view. For db.passwordSecret.name specify the referenced secret that you created in earlier steps. For example, external-db-password . For db.connectionString specify the connection string in keyword=value format, for example, host=<host> port=5432 database=stackrox user=stackrox sslmode=verify-ca For db.persistence delete the entire block. If necessary, you can specify a Certificate Authority for Central to trust the database certificate by adding a TLS block under the top-level spec, as shown in the following example: Update the central custom resource with the following configuration: spec: tls: additionalCAs: - name: db-ca content: | <certificate> central: db: isEnabled: Default 1 connectionString: "host=<host> port=5432 user=<user> sslmode=verify-ca" passwordSecret: name: external-db-password 1 You must not change the value of IsEnabled to Enabled . Click Save . Additional resources Provisioning a database in your PostgreSQL instance 1.4. Changing the subscription channel You can change the update channel for the RHACS Operator by using the OpenShift Container Platform web console or by using the command line. For upgrading to RHACS 4.0 from RHACS 3.74, you must change the update channel. Important You must change the subscription channel for all clusters where you installed the RHACS Operator, including Central and all secured clusters. Prerequisites You must verify that you are using the latest RHACS 3.74 Operator and there are no pending manual Operator upgrades. You must verify that you backed up your Central database. You have access to an OpenShift Container Platform cluster web console using an account with cluster-admin permissions. Changing the subscription channel by using the web console Use the following instructions for changing the subscription channel by using the web console: Procedure In the Administrator perspective of the OpenShift Container Platform web console, go to Operators Installed Operators . Click the RHACS Operator. Click the Subscription tab. Click the name of the update channel under Update Channel . Select stable , then click Save . For subscriptions with an Automatic approval strategy, the update begins automatically. Go back to the Operators Installed Operators page to monitor the progress of the update. When complete, the status changes to Succeeded and Up to date . For subscriptions with a Manual approval strategy, you can manually approve the update from the Subscription tab. Changing the subscription channel by using command line Use the following instructions for changing the subscription channel by using command line: Procedure Run the following command to change the subscription channel to stable : USD oc -n rhacs-operator \ 1 patch subscriptions.operators.coreos.com rhacs-operator \ --type=merge --patch='{ "spec": { "channel": "stable" }}' 1 If you use Kubernetes, enter kubectl instead of oc . During the update, the RHACS Operator provisions a new deployment called central-db and your data begins migrating. It takes around 30 minutes and happens only after you upgrade. 1.5. Rolling back an Operator upgrade To roll back an Operator upgrade, you must perform the steps described in one of the following sections. You can roll back an Operator upgrade by using the CLI or the OpenShift Container Platform web console. Note If you are rolling back from RHACS 4.0, you can only rollback to the latest patch release version of RHACS 3.74. 1.5.1. Rolling back an Operator upgrade by using the CLI You can roll back the Operator version by using CLI commands. Procedure Delete the OLM subscription by running the following command: For OpenShift Container Platform, run the following command: USD oc -n rhacs-operator delete subscription rhacs-operator For Kubernetes, run the following command: USD kubectl -n rhacs-operator delete subscription rhacs-operator Delete the cluster service version (CSV) by running the following command: For OpenShift Container Platform, run the following command: USD oc -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operator For Kubernetes, run the following command: USD kubectl -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operator Determine the version you want to roll back to by choosing one of the following options: If the current Central instance is running, query the RHACS API to get the rollback version by running the following command: USD curl -k -s -u <user>:<password> https://<central hostname>/v1/centralhealth/upgradestatus | jq -r .upgradeStatus.forceRollbackTo If the current Central instance is not running, perform the following steps: Note This procedure can only be used for RHACS release 3.74 and earlier when the rocksdb database is installed. Ensure the Central deployment is scaled down by running the following command: For OpenShift Container Platform, run the following command: USD oc scale -n <central namespace> -replicas=0 deploy/central For Kubernetes, run the following command: USD kubectl scale -n <central namespace> -replicas=0 deploy/central Save the following pod spec as a YAML file: apiVersion: v1 kind: Pod metadata: name: get--db-version spec: containers: - name: get--db-version image: registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<rollback version> command: - sh args: - '-c' - "cat /var/lib/stackrox/./migration_version.yaml | grep '^image:' | cut -f 2 -d : | tr -d ' '" volumeMounts: - name: stackrox-db mountPath: /var/lib/stackrox volumes: - name: stackrox-db persistentVolumeClaim: claimName: stackrox-db Create a pod in your Central namespace by running the following command using the YAML file that you saved: For OpenShift Container Platform, run the following command: USD oc create -n <central namespace> -f pod.yaml For Kubernetes, run the following command: USD kubectl create -n <central namespace> -f pod.yaml After pod creation is complete, get the version by running the following command: For OpenShift Container Platform, run the following command: USD oc logs -n <central namespace> get--db-version For Kubernetes, run the following command: USD kubectl logs -n <central namespace> get--db-version Edit the central-config.yaml ConfigMap to set the maintenance.forceRollBackVersion:<version> parameter by running the following command: For OpenShift Container Platform, run the following command: USD oc get configmap -n <central namespace> central-config -o yaml | sed -e "s/forceRollbackVersion: none/forceRollbackVersion: <version>/" | oc -n <central namespace> apply -f - For Kubernetes, run the following command: USD kubectl get configmap -n <central namespace> central-config -o yaml | sed -e "s/forceRollbackVersion: none/forceRollbackVersion: <version>/" | kubectl -n <central namespace> apply -f - Set the image for the Central deployment using the version string shown in Step 3 as the image tag. For example, run the following command: For OpenShift Container Platform, run the following command: USD oc set image -n <central namespace> deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<version> For Kubernetes, run the following command: USD kubectl set image -n <central namespace> deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<version> Verification Ensure that the Central pod starts and has a ready status. If the pod crashes, check the logs to see if the backup was restored. A successful log message appears similar to the following example: Reinstall the Operator on the rolled back channel. For example, 3.74.2 is installed on the rhacs-3.74 channel. 1.5.2. Rolling back an Operator upgrade by using the web console You can roll back the Operator version by using the OpenShift Container Platform web console. Prerequisites You have access to an OpenShift Container Platform cluster web console using an account with cluster-admin permissions. Procedure Go to the Operators Installed Operators page. Click the RHACS Operator. On the Operator Details page, select Uninstall Operator from the Actions list. Following this action, the Operator stops running and no longer receives updates. Determine the version you want to roll back to by choosing one of the following options: If the current Central instance is running, you can query the RHACS API to get the rollback version by running the following command from a terminal window: USD curl -k -s -u <user>:<password> https://<central hostname>/v1/centralhealth/upgradestatus | jq -r .upgradeStatus.forceRollbackTo You can create a pod and extract the version by performing the following steps: Note This procedure can only be used for RHACS release 3.74 and earlier when the rocksdb database is installed. Go to Workloads Deployments central . Under Deployment details , click the down arrow to the pod count to scale down the pod. Go to Workloads Pods Create Pod and paste the contents of the pod spec as shown in the following example into the editor: apiVersion: v1 kind: Pod metadata: name: get--db-version spec: containers: - name: get--db-version image: registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<rollback version> command: - sh args: - '-c' - "cat /var/lib/stackrox/./migration_version.yaml | grep '^image:' | cut -f 2 -d : | tr -d ' '" volumeMounts: - name: stackrox-db mountPath: /var/lib/stackrox volumes: - name: stackrox-db persistentVolumeClaim: claimName: stackrox-db Click Create . After the pod is created, click the Logs tab to get the version string. Update the rollback configuration by performing the following steps: Go to Workloads ConfigMaps central-config and select Edit ConfigMap from the Actions list. Find the forceRollbackVersion line in the value of the central-config.yaml key. Replace none with 3.73.3 , and then save the file. Update Central to the earlier version by performing the following steps: Go to Workloads Deployments central and select Edit Deployment from the Actions list. Update the image name, and then save the changes. Verification Ensure that the Central pod starts and has a ready status. If the pod crashes, check the logs to see if the backup was restored. A successful log message appears similar to the following example: Reinstall the Operator on the rolled back channel. For example, 3.74.2 is installed on the rhacs-3.74 channel. Additional resources Installing Central using the Operator method Operator Lifecycle Manager workflow Manually approving a pending Operator update 1.6. Troubleshooting Operator upgrade issues Follow these instructions to investigate and resolve upgrade-related issues for the RHACS Operator. 1.6.1. Central DB cannot be scheduled Follow the instructions here to troubleshoot a failing Central DB pod during an upgrade: Check the status of the central-db pod: USD oc -n <namespace> get pod -l app=central-db 1 1 If you use Kubernetes, enter kubectl instead of oc . If the status of the pod is Pending , use the describe command to get more details: USD oc -n <namespace> describe po/<central-db-pod-name> 1 1 If you use Kubernetes, enter kubectl instead of oc . You might see the FailedScheduling warning message: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 54s default-scheduler 0/7 nodes are available: 1 Insufficient memory, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 Insufficient cpu. preemption: 0/7 nodes are available: 3 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod. This warning message suggests that the scheduled node had insufficient memory to accommodate the pod's resource requirements. If you have a small environment, consider increasing resources on the nodes or adding a larger node that can support the database. Otherwise, consider decreasing the resource requirements for the central-db pod in the custom resource under central db resources . However, running central with fewer resources than the recommended minimum might lead to degraded performance for RHACS. 1.6.2. Central or Secured cluster fails to deploy When RHACS Operator has the following conditions, you must check the custom resource conditions to find the issue: If the Operator fails to deploy Central or Secured Cluster If the Operator fails to apply CR changes to actual resources For Central, run the following command to check the conditions: USD oc -n rhacs-operator describe centrals.platform.stackrox.io 1 1 If you use Kubernetes, enter kubectl instead of oc . For Secured clusters, run the following command to check the conditions: USD oc -n rhacs-operator describe securedclusters.platform.stackrox.io 1 1 If you use Kubernetes, enter kubectl instead of oc . You can identify configuration errors from the conditions output: Example output Conditions: Last Transition Time: 2023-04-19T10:49:57Z Status: False Type: Deployed Last Transition Time: 2023-04-19T10:49:57Z Status: True Type: Initialized Last Transition Time: 2023-04-19T10:59:10Z Message: Deployment.apps "central" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "50": must be less than or equal to cpu limit Reason: ReconcileError Status: True Type: Irreconcilable Last Transition Time: 2023-04-19T10:49:57Z Message: No proxy configuration is desired Reason: NoProxyConfig Status: False Type: ProxyConfigFailed Last Transition Time: 2023-04-19T10:49:57Z Message: Deployment.apps "central" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "50": must be less than or equal to cpu limit Reason: InstallError Status: True Type: ReleaseFailed Additionally, you can view RHACS pod logs to find more information about the issue. Run the following command to view the logs: oc -n rhacs-operator logs deploy/rhacs-operator-controller-manager manager 1 1 If you use Kubernetes, enter kubectl instead of oc .

Commands:
spec: central: db: isEnabled: Default 1 persistence: persistentVolumeClaim: 2 claimName: central-db size: 100Gi storageClassName: <storage-class-name>
oc create secret generic external-db-password \ 1 --from-file=password=<password.txt> 2
spec: tls: additionalCAs: - name: db-ca content: | <certificate> central: db: isEnabled: Default 1 connectionString: "host=<host> port=5432 user=<user> sslmode=verify-ca" passwordSecret: name: external-db-password
oc -n rhacs-operator \ 1 patch subscriptions.operators.coreos.com rhacs-operator --type=merge --patch='{ "spec": { "channel": "stable" }}'
oc -n rhacs-operator delete subscription rhacs-operator
kubectl -n rhacs-operator delete subscription rhacs-operator
oc -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operator
kubectl -n rhacs-operator delete csv -l operators.coreos.com/rhacs-operator.rhacs-operator
curl -k -s -u <user>:<password> https://<central hostname>/v1/centralhealth/upgradestatus | jq -r .upgradeStatus.forceRollbackTo
oc scale -n <central namespace> -replicas=0 deploy/central
kubectl scale -n <central namespace> -replicas=0 deploy/central
apiVersion: v1 kind: Pod metadata: name: get-previous-db-version spec: containers: - name: get-previous-db-version image: registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<rollback version> command: - sh args: - '-c' - "cat /var/lib/stackrox/.previous/migration_version.yaml | grep '^image:' | cut -f 2 -d : | tr -d ' '" volumeMounts: - name: stackrox-db mountPath: /var/lib/stackrox volumes: - name: stackrox-db persistentVolumeClaim: claimName: stackrox-db
oc create -n <central namespace> -f pod.yaml
kubectl create -n <central namespace> -f pod.yaml
oc logs -n <central namespace> get-previous-db-version
kubectl logs -n <central namespace> get-previous-db-version
oc get configmap -n <central namespace> central-config -o yaml | sed -e "s/forceRollbackVersion: none/forceRollbackVersion: <version>/" | oc -n <central namespace> apply -f -
kubectl get configmap -n <central namespace> central-config -o yaml | sed -e "s/forceRollbackVersion: none/forceRollbackVersion: <version>/" | kubectl -n <central namespace> apply -f -
oc set image -n <central namespace> deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<version>
kubectl set image -n <central namespace> deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<version>
Clone to Migrate ".previous", ""
curl -k -s -u <user>:<password> https://<central hostname>/v1/centralhealth/upgradestatus | jq -r .upgradeStatus.forceRollbackTo
apiVersion: v1 kind: Pod metadata: name: get-previous-db-version spec: containers: - name: get-previous-db-version image: registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:<rollback version> command: - sh args: - '-c' - "cat /var/lib/stackrox/.previous/migration_version.yaml | grep '^image:' | cut -f 2 -d : | tr -d ' '" volumeMounts: - name: stackrox-db mountPath: /var/lib/stackrox volumes: - name: stackrox-db persistentVolumeClaim: claimName: stackrox-db
Clone to Migrate ".previous", ""
oc -n <namespace> get pod -l app=central-db 1
oc -n <namespace> describe po/<central-db-pod-name> 1
Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 54s default-scheduler 0/7 nodes are available: 1 Insufficient memory, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 4 Insufficient cpu. preemption: 0/7 nodes are available: 3 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.
oc -n rhacs-operator describe centrals.platform.stackrox.io 1
oc -n rhacs-operator describe securedclusters.platform.stackrox.io 1
Conditions: Last Transition Time: 2023-04-19T10:49:57Z Status: False Type: Deployed Last Transition Time: 2023-04-19T10:49:57Z Status: True Type: Initialized Last Transition Time: 2023-04-19T10:59:10Z Message: Deployment.apps "central" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "50": must be less than or equal to cpu limit Reason: ReconcileError Status: True Type: Irreconcilable Last Transition Time: 2023-04-19T10:49:57Z Message: No proxy configuration is desired Reason: NoProxyConfig Status: False Type: ProxyConfigFailed Last Transition Time: 2023-04-19T10:49:57Z Message: Deployment.apps "central" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: "50": must be less than or equal to cpu limit Reason: InstallError Status: True Type: ReleaseFailed
-n rhacs-operator logs deploy/rhacs-operator-controller-manager manager 1