Chapter 8. Upgrading Service Telemetry Framework to version 1.5 To upgrade Service Telemetry Framework (STF) 1.4 to STF 1.5, you must complete the following steps: Replace AMQ Certificate Manager with Certificate Manager. Remove the ClusterServiceVersion and Subscription objects for Smart Gateway Operator and Service Telemetry Operator in the service-telemetry namespace on your Red Hat OpenShift Container Platform environment. Upgrade Red Hat OpenShift Container Platform from 4.8 to 4.10. Re-enable the operators that you removed. Update the AMQ Interconnect CA Certificate on Red Hat OpenStack Platform (RHOSP). Prerequisites You have backed up your data. There is an outage during the Red Hat OpenShift Container Platform upgrade. You cannot reconfigure the ServiceTelemetry and SmartGateway objects during the Operators replacement. You have prepared your environment for upgrade from Red Hat OpenShift Container Platform 4.8 to the supported version, 4.10. The Red Hat OpenShift Container Platform cluster is fully-connected. STF does not support disconnected or restricted-network clusters. 8.1. Removing the Service Telemetry Framework 1.4 Operators Remove the Service Telemetry Framework (STF) 1.4 Operators and the AMQ Certificate Manager Operator from the Red Hat OpenShift Container Platform 4.8. Procedure Remove the Service Telemetry Operator. Remove the Smart Gateway Operator. Remove the AMQ Certificate Manager Operator. Remove the Grafana Operator. Additional resources For more information about removing Operators from the Red Hat OpenShift Container Platform, see Deleting Operators from a cluster . 8.1.1. Removing the Service Telemetry Operator As part of upgrading your Service Telemetry Framework (STF) installation, you must remove the Service Telemetry Operator in the service-telemetry namespace on your Red Hat OpenShift Container Platform environment. Procedure Change to the service-telemetry project: USD oc project service-telemetry Remove the Service Telemetry Operator Subscription: USD oc delete sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry subscription.operators.coreos.com "service-telemetry-operator" deleted Remove the Service Telemetry Operator ClusterServiceVersion : USD oc delete csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry clusterserviceversion.operators.coreos.com "service-telemetry-operator.v1.4.1669718959" deleted Verification Verify that the Service Telemetry Operator deployment is not running: USD oc get deploy --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace. Verify the Service Telemetry Operator subscription is absent: USD oc get sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace. Verify the Service Telemetry Operator ClusterServiceVersion is absent: USD oc get csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace. 8.1.2. Removing the Smart Gateway Operator As part of upgrading your Service Telemetry Framework (STF) installation, you must remove the Smart Gateway Operator in the service-telemetry namespace on your Red Hat OpenShift Container Platform environment. Procedure Change to the service-telemetry project: USD oc project service-telemetry Remove the Smart Gateway Operator Subscription: USD oc delete sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry subscription.operators.coreos.com "smart-gateway-operator-stable-1.4-redhat-operators-openshift-marketplace" deleted Remove the Smart Gateway Operator ClusterServiceVersion : USD oc delete csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry clusterserviceversion.operators.coreos.com "smart-gateway-operator.v4.0.1669718962" deleted Verification Verify that the Smart Gateway Operator deployment is not running: USD oc get deploy --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace. Verify the Smart Gateway Operator subscription is absent: USD oc get sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace. Verify the Smart Gateway Operator ClusterServiceVersion is absent: USD oc get csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace. 8.1.3. Removing the AMQ Certificate Manager Operator Procedure Remove the AMQ Certificate Manager Operator Subscription: USD oc delete sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators subscription.operators.coreos.com "amq7-cert-manager-operator" deleted Remove the AMQ Certificate Manager Operator ClusterServiceVersion : USD oc delete csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators clusterserviceversion.operators.coreos.com "amq7-cert-manager.v1.0.11" deleted Verification Verify that the AMQ Certificate Manager Operator deployment is not running: USD oc get deploy --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators No resources found in openshift-operators namespace. Verify that the AMQ Certificate Manager Operator subscription is absent: USD oc get sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.service-telemetry No resources found in openshift-operators namespace. Verify that the AMQ Certificate Manager Operator Cluster Service Version is absent: USD oc get csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators No resources found in openshift-operators namespace. 8.1.4. Removing the Grafana Operator Procedure Remove the Grafana Operator Subscription: USD oc delete sub --selector=operators.coreos.com/grafana-operator.service-telemetry subscription.operators.coreos.com "grafana-operator" deleted Remove the Grafana Operator ClusterServiceVersion : USD oc delete csv --selector=operators.coreos.com/grafana-operator.service-telemetry clusterserviceversion.operators.coreos.com "grafana-operator.v3.10.3" deleted Verification Verify the Grafana Operator deployment is not running: USD oc get deploy --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace. Verify the Grafana Operator subscription is absent: USD oc get sub --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace. Verify the Grafana Operator Cluster Service Version is absent: USD oc get csv --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace. 8.2. Upgrading Red Hat OpenShift Container Platform to 4.10 Service Telemetry Framework (STF) 1.5 is only compatible with Red Hat OpenShift Container Platform 4.10. For more information about upgrading your Red Hat OpenShift Container Platform from 4.8 to 4.10, see Updating clusters overview . Prerequisites You removed the STF 1.4 Operators. You removed the AMQ Certificate Manager Operator and Grafana Operator. You must remove the Operators before you upgrade Red Hat OpenShift Container Platform because the Operator APIs are incompatible with 4.10. For more information about preparing your Red Hat OpenShift Container Platform for upgrade from 4.8 to 4.10, see Understanding OpenShift Container Platform updates . Verify the suitability of your Red Hat OpenShift Container Platform upgrade: USD oc adm upgrade You cannot upgrade the cluster if you encounter the following error message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: service-telemetry/grafana-operator.v3.10.3 is incompatible with OpenShift minor versions greater than 4.8,openshift-operators/amq7-cert-manager.v1.0.11 is incompatible with OpenShift minor versions greater than 4.8 8.3. Installing the Service Telemetry Framework 1.5 Operators Install the Service Telemetry Framework (STF) 1.5 Operators and the Certificate Manager for OpenShift Operator on your Red Hat OpenShift Container Platform environment. See Section 1.1, "Support for Service Telemetry Framework" for more information about STF support status and life cycle. Note After a successful STF 1.5 install, you must retrieve and apply the AMQ Interconnect CA certificate to the Red Hat OpenStack Platform environment, or the transport layer and telemetry data becomes unavailable. For more information about updating the AMQ Interconnect CA certificate, see Section 8.4, "Updating the AMQ Interconnect CA Certificate on Red Hat OpenStack Platform" . Prerequisites You have upgraded your Red Hat OpenShift Container Platform environment to 4.10. For more information about upgrading Red Hat OpenShift Container Platform, see Section 8.2, "Upgrading Red Hat OpenShift Container Platform to 4.10" . Your Red Hat OpenShift Container Platform environment network is fully-connected. Procedure Change to the service-telemetry project: USD oc project service-telemetry Create a namespace for the cert-manager Operator: USD oc create -f - <<EOF apiVersion: project.openshift.io/v1 kind: Project metadata: name: openshift-cert-manager-operator spec: finalizers: - kubernetes EOF Create an OperatorGroup for the cert-manager Operator: USD oc create -f - <<EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: {} EOF Subscribe to the cert-manager Operator with the redhat-operators CatalogSource: USD oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: channel: tech-preview installPlanApproval: Automatic name: openshift-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF Validate your ClusterServiceVersion . Ensure that the phase of cert-manager Operator is Succeeded : USD oc get csv --namespace openshift-cert-manager-operator --selector=operators.coreos.com/openshift-cert-manager-operator.openshift-cert-manager-operator NAME DISPLAY VERSION REPLACES PHASE openshift-cert-manager.v1.7.1 cert-manager Operator for Red Hat OpenShift 1.7.1-1 Succeeded Optional: Resubscribe to the Grafana Operator. For more information, see: test Section 5.1.1, "Configuring Grafana to host the dashboard" . Create the Service Telemetry Operator subscription to manage the STF instances: USD oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: service-telemetry-operator namespace: service-telemetry spec: channel: stable-1.5 installPlanApproval: Automatic name: service-telemetry-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF Validate the Service Telemetry Operator and the dependent operators: USD oc get csv --namespace service-telemetry NAME DISPLAY VERSION REPLACES PHASE amq7-interconnect-operator.v1.10.13 Red Hat Integration - AMQ Interconnect 1.10.13 amq7-interconnect-operator.v1.10.4 Succeeded elasticsearch-eck-operator-certified.v2.5.0 Elasticsearch (ECK) Operator 2.5.0 elasticsearch-eck-operator-certified.v2.4.0 Succeeded openshift-cert-manager.v1.7.1 cert-manager Operator for Red Hat OpenShift 1.7.1-1 Succeeded prometheusoperator.0.47.0 Prometheus Operator 0.47.0 prometheusoperator.0.37.0 Succeeded service-telemetry-operator.v1.5.1669950702 Service Telemetry Operator 1.5.1669950702 Succeeded smart-gateway-operator.v5.0.1669950681 Smart Gateway Operator 5.0.1669950681 Succeeded Verification Verify that the Service Telemetry Operator has successfully reconciled. USD oc logs -f --selector=name=service-telemetry-operator [...] ----- Ansible Task Status Event StdOut (infra.watch/v1beta1, Kind=ServiceTelemetry, default/service-telemetry) ----- PLAY RECAP ********************************************************************* localhost : ok=115 changed=0 unreachable=0 failed=0 skipped=21 rescued=0 ignored=0 USD oc get pods NAME READY STATUS RESTARTS AGE alertmanager-default-0 3/3 Running 0 20h default-cloud1-ceil-event-smartgateway-6d57ffbbdc-5mrj8 2/2 Running 1 (3m42s ago) 4m21s default-cloud1-ceil-meter-smartgateway-67684d88c8-62mp7 3/3 Running 1 (3m43s ago) 4m20s default-cloud1-coll-event-smartgateway-66cddd5567-qb6p2 2/2 Running 1 (3m42s ago) 4m19s default-cloud1-coll-meter-smartgateway-76d5ff6db5-z5ch8 3/3 Running 0 4m20s default-cloud1-sens-meter-smartgateway-7b59669fdd-c42zg 3/3 Running 1 (3m43s ago) 4m20s default-interconnect-845c4b647c-wwfcm 1/1 Running 0 4m10s elastic-operator-57b57964c5-6q88r 1/1 Running 8 (17h ago) 20h elasticsearch-es-default-0 1/1 Running 0 21h grafana-deployment-59c54f7d7c-zjnhm 2/2 Running 0 20h interconnect-operator-848889bf8b-bq2zx 1/1 Running 0 20h prometheus-default-0 3/3 Running 1 (20h ago) 20h prometheus-operator-5d7b69fd46-c2xlv 1/1 Running 0 20h service-telemetry-operator-79fb664dfb-nj8jn 1/1 Running 0 5m11s smart-gateway-operator-79557664f8-ql7xr 1/1 Running 0 5m7s 8.4. Updating the AMQ Interconnect CA Certificate on Red Hat OpenStack Platform After you upgrade to Service Telemetry Framework (STF) v1.5, the CA certificate for AMQ Interconnect regenerates. In STF v1.4 the CA certificate for AMQ Interconnect is valid for three months and must be updated periodically in Red Hat OpenStack Platform (RHOSP). In STF v1.5, the generated certificates are valid for the lifetime of the RHOSP lifecycle, 70080 hours by default. Prerequisites You have successfully installed STF v1.5 and updated the CA certificate for AMQ Interconnect. Procedure For more information about how to update the CA certificate in your RHOSP environment, see Chapter 6, Renewing the AMQ Interconnect certificate

Commands:
oc project service-telemetry
oc delete sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry subscription.operators.coreos.com "service-telemetry-operator" deleted
oc delete csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry clusterserviceversion.operators.coreos.com "service-telemetry-operator.v1.4.1669718959" deleted
oc get deploy --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace.
oc get sub --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace.
oc get csv --selector=operators.coreos.com/service-telemetry-operator.service-telemetry No resources found in service-telemetry namespace.
oc project service-telemetry
oc delete sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry subscription.operators.coreos.com "smart-gateway-operator-stable-1.4-redhat-operators-openshift-marketplace" deleted
oc delete csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry clusterserviceversion.operators.coreos.com "smart-gateway-operator.v4.0.1669718962" deleted
oc get deploy --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace.
oc get sub --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace.
oc get csv --selector=operators.coreos.com/smart-gateway-operator.service-telemetry No resources found in service-telemetry namespace.
oc delete sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators subscription.operators.coreos.com "amq7-cert-manager-operator" deleted
oc delete csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators clusterserviceversion.operators.coreos.com "amq7-cert-manager.v1.0.11" deleted
oc get deploy --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators No resources found in openshift-operators namespace.
oc get sub --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.service-telemetry No resources found in openshift-operators namespace.
oc get csv --namespace openshift-operators --selector=operators.coreos.com/amq7-cert-manager-operator.openshift-operators No resources found in openshift-operators namespace.
oc delete sub --selector=operators.coreos.com/grafana-operator.service-telemetry subscription.operators.coreos.com "grafana-operator" deleted
oc delete csv --selector=operators.coreos.com/grafana-operator.service-telemetry clusterserviceversion.operators.coreos.com "grafana-operator.v3.10.3" deleted
oc get deploy --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace.
oc get sub --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace.
oc get csv --selector=operators.coreos.com/grafana-operator.service-telemetry No resources found in service-telemetry namespace.
oc adm upgrade
Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: service-telemetry/grafana-operator.v3.10.3 is incompatible with OpenShift minor versions greater than 4.8,openshift-operators/amq7-cert-manager.v1.0.11 is incompatible with OpenShift minor versions greater than 4.8
oc project service-telemetry
oc create -f - <<EOF apiVersion: project.openshift.io/v1 kind: Project metadata: name: openshift-cert-manager-operator spec: finalizers: - kubernetes EOF
oc create -f - <<EOF apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: {} EOF
oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-cert-manager-operator namespace: openshift-cert-manager-operator spec: channel: tech-preview installPlanApproval: Automatic name: openshift-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
oc get csv --namespace openshift-cert-manager-operator --selector=operators.coreos.com/openshift-cert-manager-operator.openshift-cert-manager-operator NAME DISPLAY VERSION REPLACES PHASE openshift-cert-manager.v1.7.1 cert-manager Operator for Red Hat OpenShift 1.7.1-1 Succeeded
oc create -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: service-telemetry-operator namespace: service-telemetry spec: channel: stable-1.5 installPlanApproval: Automatic name: service-telemetry-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
oc get csv --namespace service-telemetry NAME DISPLAY VERSION REPLACES PHASE amq7-interconnect-operator.v1.10.13 Red Hat Integration - AMQ Interconnect 1.10.13 amq7-interconnect-operator.v1.10.4 Succeeded elasticsearch-eck-operator-certified.v2.5.0 Elasticsearch (ECK) Operator 2.5.0 elasticsearch-eck-operator-certified.v2.4.0 Succeeded openshift-cert-manager.v1.7.1 cert-manager Operator for Red Hat OpenShift 1.7.1-1 Succeeded prometheusoperator.0.47.0 Prometheus Operator 0.47.0 prometheusoperator.0.37.0 Succeeded service-telemetry-operator.v1.5.1669950702 Service Telemetry Operator 1.5.1669950702 Succeeded smart-gateway-operator.v5.0.1669950681 Smart Gateway Operator 5.0.1669950681 Succeeded
oc logs -f --selector=name=service-telemetry-operator [...] ----- Ansible Task Status Event StdOut (infra.watch/v1beta1, Kind=ServiceTelemetry, default/service-telemetry) ----- PLAY RECAP ********************************************************************* localhost : ok=115 changed=0 unreachable=0 failed=0 skipped=21 rescued=0 ignored=0 oc get pods NAME READY STATUS RESTARTS AGE alertmanager-default-0 3/3 Running 0 20h default-cloud1-ceil-event-smartgateway-6d57ffbbdc-5mrj8 2/2 Running 1 (3m42s ago) 4m21s default-cloud1-ceil-meter-smartgateway-67684d88c8-62mp7 3/3 Running 1 (3m43s ago) 4m20s default-cloud1-coll-event-smartgateway-66cddd5567-qb6p2 2/2 Running 1 (3m42s ago) 4m19s default-cloud1-coll-meter-smartgateway-76d5ff6db5-z5ch8 3/3 Running 0 4m20s default-cloud1-sens-meter-smartgateway-7b59669fdd-c42zg 3/3 Running 1 (3m43s ago) 4m20s default-interconnect-845c4b647c-wwfcm 1/1 Running 0 4m10s elastic-operator-57b57964c5-6q88r 1/1 Running 8 (17h ago) 20h elasticsearch-es-default-0 1/1 Running 0 21h grafana-deployment-59c54f7d7c-zjnhm 2/2 Running 0 20h interconnect-operator-848889bf8b-bq2zx 1/1 Running 0 20h prometheus-default-0 3/3 Running 1 (20h ago) 20h prometheus-operator-5d7b69fd46-c2xlv 1/1 Running 0 20h service-telemetry-operator-79fb664dfb-nj8jn 1/1 Running 0 5m11s smart-gateway-operator-79557664f8-ql7xr 1/1 Running 0 5m7s