Chapter 5. Using operational features of Service Telemetry Framework You can use the following operational features to provide additional functionality to the Service Telemetry Framework (STF): Configuring dashboards Configuring the metrics retention time period Configuring alerts Configuring SNMP traps Configuring high availability Configuring an alternate observability strategy Monitoring the resource use of OpenStack services Monitoring container health and API status 5.1. Dashboards in Service Telemetry Framework Use the third-party application, Grafana, to visualize system-level metrics that the data collectors collectd and Ceilometer gather for each individual host node. For more information about configuring data collectors, see Section 4.1, "Deploying Red Hat OpenStack Platform overcloud for Service Telemetry Framework using director" . You can use dashboards to monitor a cloud: Infrastructure dashboard Use the infrastructure dashboard to view metrics for a single node at a time. Select a node from the upper left corner of the dashboard. Cloud view dashboard Use the cloud view dashboard to view panels to monitor service resource usage, API stats, and cloud events. You must enable API health monitoring and service monitoring to provide the data for this dashboard. API health monitoring is enabled by default in the STF base configuration. For more information, see Section 4.1.3, "Creating the base configuration for STF" . For more information about API health monitoring, see Section 5.9, "Red Hat OpenStack Platform API status and containerized services health" . For more information about RHOSP service monitoring, see Section 5.8, "Resource usage of Red Hat OpenStack Platform services" . Virtual machine view dashboard Use the virtual machine view dashboard to view panels to monitor virtual machine infrastructure usage. Select a cloud and project from the upper left corner of the dashboard. You must enable event storage if you want to enable the event annotations on this dashboard. For more information, see Section 3.2, "Creating a ServiceTelemetry object in Red Hat OpenShift Container Platform" . Memcached view dashboard Use the memcached view dashboard to view panels to monitor connections, availability, system metrics and cache performance. Select a cloud from the upper left corner of the dashboard. 5.1.1. Configuring Grafana to host the dashboard Grafana is not included in the default Service Telemetry Framework (STF) deployment, so you must deploy the Grafana Operator from community-operators CatalogSource. If you use the Service Telemetry Operator to deploy Grafana, it results in a Grafana instance and the configuration of the default data sources for the local STF deployment. Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Subscribe to the Grafana Operator by using the community-operators CatalogSource: Warning Community Operators are Operators which have not been vetted or verified by Red Hat. Community Operators should be used with caution because their stability is unknown. Red Hat provides no support for community Operators. Learn more about Red Hat's third party software support policy USD oc apply -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: grafana-operator namespace: service-telemetry spec: channel: v4 installPlanApproval: Automatic name: grafana-operator source: community-operators sourceNamespace: openshift-marketplace EOF Verify that the Operator launched successfully. In the command output, if the value of the PHASE column is Succeeded , the Operator launched successfully: USD oc get csv --selector operators.coreos.com/grafana-operator.service-telemetry NAME DISPLAY VERSION REPLACES PHASE grafana-operator.v4.10.1 Grafana Operator 4.10.1 grafana-operator.v4.10.0 Succeeded To launch a Grafana instance, create or modify the ServiceTelemetry object. Set graphing.enabled and graphing.grafana.ingressEnabled to true . Optionally, set the value of graphing.grafana.baseImage to the Grafana workload container image that will be deployed: USD oc edit stf default apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry ... spec: ... graphing: enabled: true grafana: ingressEnabled: true baseImage: 'registry.redhat.io/rhel8/grafana:7' Verify that the Grafana instance deployed: USD oc get pod -l app=grafana NAME READY STATUS RESTARTS AGE grafana-deployment-7fc7848b56-sbkhv 1/1 Running 0 1m Verify that the Grafana data sources installed correctly: USD oc get grafanadatasources NAME AGE default-datasources 20h Verify that the Grafana route exists: USD oc get route grafana-route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD grafana-route grafana-route-service-telemetry.apps.infra.watch grafana-service 3000 edge None 5.1.2. Overriding the default Grafana container image The dashboards in Service Telemetry Framework (STF) require features that are available only in Grafana version 8.1.0 and later. By default, the Service Telemetry Operator installs a compatible version. You can override the base Grafana image by specifying the image path to an image registry with graphing.grafana.baseImage . Procedure Ensure that you have the correct version of Grafana: USD oc get pod -l "app=grafana" -ojsonpath='{.items[0].spec.containers[0].image}' docker.io/grafana/grafana:7.3.10 If the running image is older than 8.1.0, patch the ServiceTelemetry object to update the image. Service Telemetry Operator updates the Grafana manifest, which restarts the Grafana deployment: USD oc patch stf/default --type merge -p '{"spec":{"graphing":{"grafana":{"baseImage":"docker.io/grafana/grafana:8.1.5"}}}}' Verify that a new Grafana pod exists and has a STATUS value of Running : USD oc get pod -l "app=grafana" NAME READY STATUS RESTARTS AGE grafana-deployment-fb9799b58-j2hj2 1/1 Running 0 10s Verify that the new instance is running the updated image: USD oc get pod -l "app=grafana" -ojsonpath='{.items[0].spec.containers[0].image}' docker.io/grafana/grafana:8.1.0 5.1.3. Importing dashboards The Grafana Operator can import and manage dashboards by creating GrafanaDashboard objects. You can view example dashboards at https://github.com/infrawatch/dashboards . Procedure Import the infrastructure dashboard: USD oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-dashboard.yaml grafanadashboard.integreatly.org/rhos-dashboard-1 created Import the cloud dashboard: Warning In the stf-connectors.yaml file, ensure you set the value of the collectd virt plugin parameter hostname_format to name uuid hostname , otherwise some of the panels on the cloud dashboard display no information. For more information about the virt plugin, see collectd plugins . USD oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-cloud-dashboard.yaml grafanadashboard.integreatly.org/rhos-cloud-dashboard-1 created Import the cloud events dashboard: USD oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-cloudevents-dashboard.yaml grafanadashboard.integreatly.org/rhos-cloudevents-dashboard created Import the virtual machine dashboard: USD oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/virtual-machine-view.yaml grafanadashboard.integreatly.org/virtual-machine-view-1 configured Import the memcached dashboard: USD oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/memcached-dashboard.yaml grafanadashboard.integreatly.org/memcached-dashboard-1 created Verify that the dashboards are available: USD oc get grafanadashboards NAME AGE memcached-dashboard-1 7s rhos-cloud-dashboard-1 23s rhos-cloudevents-dashboard 18s rhos-dashboard-1 29s virtual-machine-view-1 13s Retrieve the Grafana route address: USD oc get route grafana-route -ojsonpath='{.spec.host}' grafana-route-service-telemetry.apps.infra.watch In a web browser, navigate to https:// <grafana_route_address> . Replace <grafana_route_address> with the value that you retrieved in the step. To view the dashboard, click Dashboards and Manage . 5.1.4. Retrieving and setting Grafana login credentials When Grafana is enabled, you can login using openshift authentication, or the default username and password set by the Grafana Operator. You can override the credentials in the ServiceTelemetry object to have Service Telemetry Framework (STF) set the username and password for Grafana instead. Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Retrieve the existing username and password from the STF object: USD oc get stf default -o jsonpath="{.spec.graphing.grafana['adminUser','adminPassword']}" To modify the default values of the Grafana administrator username and password through the ServiceTelemetry object, use the graphing.grafana.adminUser and graphing.grafana.adminPassword parameters. USD oc edit stf default Wait for the grafana pod to restart with the new credentials in place USD oc get po -l app=grafana -w 5.2. Metrics retention time period in Service Telemetry Framework The default retention time for metrics stored in Service Telemetry Framework (STF) is 24 hours, which provides enough data for trends to develop for the purposes of alerting. For long-term storage, use systems designed for long-term data retention, for example, Thanos. Additional resources To adjust STF for additional metrics retention time, see Section 5.2.1, "Editing the metrics retention time period in Service Telemetry Framework" . For recommendations about Prometheus data storage and estimating storage space, see https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects For more information about Thanos, see https://thanos.io/ 5.2.1. Editing the metrics retention time period in Service Telemetry Framework You can adjust Service Telemetry Framework (STF) for additional metrics retention time. Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Edit the ServiceTelemetry object: USD oc edit stf default Add retention: 7d to the storage section of backends.metrics.prometheus.storage to increase the retention period to seven days: Note If you set a long retention period, retrieving data from heavily populated Prometheus systems can result in queries returning results slowly. apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: ... backends: metrics: prometheus: enabled: true storage: strategy: persistent retention: 7d ... Save your changes and close the object. Wait for prometheus to restart with the new settings. USD oc get po -l app.kubernetes.io/name=prometheus -w Verify the new retention setting by checking the command line arguments used in the pod. USD oc describe po prometheus-default-0 | grep retention.time --storage.tsdb.retention.time=24h Additional resources For more information about the metrics retention time, see Section 5.2, "Metrics retention time period in Service Telemetry Framework" . 5.3. Alerts in Service Telemetry Framework You create alert rules in Prometheus and alert routes in Alertmanager. Alert rules in Prometheus servers send alerts to an Alertmanager, which manages the alerts. Alertmanager can silence, inhibit, or aggregate alerts, and send notifications by using email, on-call notification systems, or chat platforms. To create an alert, complete the following tasks: Create an alert rule in Prometheus. For more information, see Section 5.3.1, "Creating an alert rule in Prometheus" . Create an alert route in Alertmanager. There are two ways in which you can create an alert route: Creating a standard alert route in Alertmanager . Creating an alert route with templating in Alertmanager . Additional resources For more information about alerts or notifications with Prometheus and Alertmanager, see https://prometheus.io/docs/alerting/overview/ To view an example set of alerts that you can use with Service Telemetry Framework (STF), see https://github.com/infrawatch/service-telemetry-operator/tree/master/deploy/alerts 5.3.1. Creating an alert rule in Prometheus Prometheus evaluates alert rules to trigger notifications. If the rule condition returns an empty result set, the condition is false. Otherwise, the rule is true and it triggers an alert. Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Create a PrometheusRule object that contains the alert rule. The Prometheus Operator loads the rule into Prometheus: USD oc apply -f - <<EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: creationTimestamp: null labels: prometheus: default role: alert-rules name: prometheus-alarm-rules namespace: service-telemetry spec: groups: - name: ./openstack.rules rules: - alert: Collectd metrics receive rate is zero expr: rate(sg_total_collectd_msg_received_count[1m]) == 0 EOF To change the rule, edit the value of the expr parameter. To verify that the Operator loaded the rules into Prometheus, run the curl command against the default-prometheus-proxy route with basic authentication: USD curl -k --user "internal:USD(oc get secret default-prometheus-htpasswd -ogo-template='{{ .data.password | base64decode }}')" https://USD(oc get route default-prometheus-proxy -ogo-template='{{ .spec.host }}')/api/v1/rules {"status":"success","data":{"groups":[{"name":"./openstack.rules","file":"/etc/prometheus/rules/prometheus-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml","rules":[{"state":"inactive","name":"Collectd metrics receive count is zero","query":"rate(sg_total_collectd_msg_received_count[1m]) == 0","duration":0,"labels":{},"annotations":{},"alerts":[],"health":"ok","evaluationTime":0.00034627,"lastEvaluation":"2021-12-07T17:23:22.160448028Z","type":"alerting"}],"interval":30,"evaluationTime":0.000353787,"lastEvaluation":"2021-12-07T17:23:22.160444017Z"}]}} Additional resources For more information on alerting, see https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/alerting.md 5.3.2. Configuring custom alerts You can add custom alerts to the PrometheusRule object that you created in Section 5.3.1, "Creating an alert rule in Prometheus" . Procedure Use the oc edit command: USD oc edit prometheusrules prometheus-alarm-rules Edit the PrometheusRules manifest. Save and close the manifest. Additional resources For more information about how to configure alerting rules, see https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ . For more information about PrometheusRules objects, see https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/alerting.md 5.3.3. Creating a standard alert route in Alertmanager Use Alertmanager to deliver alerts to an external system, such as email, IRC, or other notification channel. The Prometheus Operator manages the Alertmanager configuration as a Red Hat OpenShift Container Platform secret. By default, Service Telemetry Framework (STF) deploys a basic configuration that results in no receivers: alertmanager.yaml: |- global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null' To deploy a custom Alertmanager route with STF, you must add a alertmanagerConfigManifest parameter to the Service Telemetry Operator that results in an updated secret, managed by the Prometheus Operator. Note If your alertmanagerConfigManifest contains a custom template, for example, to construct the title and text of the sent alert, you must deploy the contents of the alertmanagerConfigManifest using a base64-encoded configuration. For more information, see Section 5.3.4, "Creating an alert route with templating in Alertmanager" . Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Edit the ServiceTelemetry object for your STF deployment: USD oc edit stf default Add the new parameter alertmanagerConfigManifest and the Secret object contents to define the alertmanager.yaml configuration for Alertmanager: Note This step loads the default template that the Service Telemetry Operator manages. To verify that the changes are populating correctly, change a value, return the alertmanager-default secret, and verify that the new value is loaded into memory. For example, change the value of the parameter global.resolve_timeout from 5m to 10m . apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: backends: metrics: prometheus: enabled: true alertmanagerConfigManifest: | apiVersion: v1 kind: Secret metadata: name: 'alertmanager-default' namespace: 'service-telemetry' type: Opaque stringData: alertmanager.yaml: |- global: resolve_timeout: 10m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null' Verify that the configuration has been applied to the secret: USD oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}' global: resolve_timeout: 10m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null' Run the wget command from the prometheus pod against the alertmanager-proxy service to retrieve the status and configYAML contents, and verify that the supplied configuration matches the configuration in Alertmanager: USD oc exec -it prometheus-default-0 -c prometheus -- sh -c "wget --header \"Authorization: Bearer \USD(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -" {"status":"success","data":{"configYAML":"...",...}} Verify that the configYAML field contains the changes you expect. To clean up the environment, delete the curl pod: USD oc delete pod curl pod "curl" deleted Additional resources For more information about the Red Hat OpenShift Container Platform secret and the Prometheus operator, see Prometheus user guide on alerting . 5.3.4. Creating an alert route with templating in Alertmanager Use Alertmanager to deliver alerts to an external system, such as email, IRC, or other notification channel. The Prometheus Operator manages the Alertmanager configuration as a Red Hat OpenShift Container Platform secret. By default, Service Telemetry Framework (STF) deploys a basic configuration that results in no receivers: alertmanager.yaml: |- global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null' If the alertmanagerConfigManifest parameter contains a custom template, for example, to construct the title and text of the sent alert, you must deploy the contents of the alertmanagerConfigManifest by using a base64-encoded configuration. Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Create the necessary alertmanager config in a file called alertmanager.yaml, for example: USD cat > alertmanager.yaml <<EOF global: resolve_timeout: 10m slack_api_url: <slack_api_url> receivers: - name: slack slack_configs: - channel: #stf-alerts title: |- ... text: >- ... route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack' EOF Generate the config manifest and add it to the ServiceTelemetry object for your STF deployment: USD CONFIG_MANIFEST=USD(oc create secret --dry-run=client generic alertmanager-default --from-file=alertmanager.yaml -o json) USD oc patch stf default --type=merge -p '{"spec":{"alertmanagerConfigManifest":'"USDCONFIG_MANIFEST"'}}' Verify that the configuration has been applied to the secret: Note There will be a short delay as the operators update each object USD oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}' global: resolve_timeout: 10m slack_api_url: <slack_api_url> receivers: - name: slack slack_configs: - channel: #stf-alerts title: |- ... text: >- ... route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack' Run the wget command from the prometheus pod against the alertmanager-proxy service to retrieve the status and configYAML contents, and verify that the supplied configuration matches the configuration in Alertmanager: USD oc exec -it prometheus-default-0 -c prometheus -- /bin/sh -c "wget --header \"Authorization: Bearer \USD(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -" {"status":"success","data":{"configYAML":"...",...}} Verify that the configYAML field contains the changes you expect. Additional resources For more information about the Red Hat OpenShift Container Platform secret and the Prometheus operator, see Prometheus user guide on alerting . 5.4. Sending alerts as SNMP traps To enable SNMP traps, modify the ServiceTelemetry object and configure the snmpTraps parameters. SNMP traps are sent using version 2c. 5.4.1. Configuration parameters for snmpTraps The snmpTraps parameter contains the following sub-parameters for configuring the alert receiver: enabled Set the value of this sub-parameter to true to enable the SNMP trap alert receiver. The default value is false. target Target address to send SNMP traps. Value is a string. Default is 192.168.24.254 . port Target port to send SNMP traps. Value is an integer. Default is 162 . community Target community to send SNMP traps to. Value is a string. Default is public . retries SNMP trap retry delivery limit. Value is an integer. Default is 5 . timeout SNMP trap delivery timeout defined in seconds. Value is an integer. Default is 1 . alertOidLabel Label name in the alert that defines the OID value to send the SNMP trap as. Value is a string. Default is oid . trapOidPrefix SNMP trap OID prefix for variable bindings. Value is a string. Default is 1.3.6.1.4.1.50495.15 . trapDefaultOid SNMP trap OID when no alert OID label has been specified with the alert. Value is a string. Default is 1.3.6.1.4.1.50495.15.1.2.1 . trapDefaultSeverity SNMP trap severity when no alert severity has been set. Value is a string. Defaults to an empty string. Configure the snmpTraps parameter as part of the alerting.alertmanager.receivers definition in the ServiceTelemetry object: apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: alerting: alertmanager: receivers: snmpTraps: alertOidLabel: oid community: public enabled: true port: 162 retries: 5 target: 192.168.25.254 timeout: 1 trapDefaultOid: 1.3.6.1.4.1.50495.15.1.2.1 trapDefaultSeverity: "" trapOidPrefix: 1.3.6.1.4.1.50495.15 ... 5.4.2. Overview of the MIB definition Delivery of SNMP traps uses object identifier (OID) value 1.3.6.1.4.1.50495.15.1.2.1 by default. The management information base (MIB) schema is available at https://github.com/infrawatch/prometheus-webhook-snmp/blob/master/PROMETHEUS-ALERT-CEPH-MIB.txt . The OID number is comprised of the following component values: * The value 1.3.6.1.4.1 is a global OID defined for private enterprises. * The identifier 50495 is a private enterprise number assigned by IANA for the Ceph organization. * The other values are child OIDs of the parent. 15 prometheus objects 15.1 prometheus alerts 15.1.2 prometheus alert traps 15.1.2.1 prometheus alert trap default The prometheus alert trap default is an object comprised of several other sub-objects to OID 1.3.6.1.4.1.50495.15 which is defined by the alerting.alertmanager.receivers.snmpTraps.trapOidPrefix parameter: <trapOidPrefix>.1.1.1 alert name <trapOidPrefix>.1.1.2 status <trapOidPrefix>.1.1.3 severity <trapOidPrefix>.1.1.4 instance <trapOidPrefix>.1.1.5 job <trapOidPrefix>.1.1.6 description <trapOidPrefix>.1.1.7 labels <trapOidPrefix>.1.1.8 timestamp <trapOidPrefix>.1.1.9 rawdata The following is example output from a simple SNMP trap receiver that outputs the received trap to the console: 5.4.3. Configuring SNMP traps Prerequisites Ensure that you know the IP address or hostname of the SNMP trap receiver where you want to send the alerts to. Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry To enable SNMP traps, modify the ServiceTelemetry object: USD oc edit stf default Set the alerting.alertmanager.receivers.snmpTraps parameters: apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry ... spec: ... alerting: alertmanager: receivers: snmpTraps: enabled: true target: 10.10.10.10 Ensure that you set the value of target to the IP address or hostname of the SNMP trap receiver. Additional Information For more information about available parameters for snmpTraps , see Section 5.4.1, "Configuration parameters for snmpTraps" . 5.4.4. Creating alerts for SNMP traps You can create alerts that are configured for delivery by SNMP traps by adding labels that are parsed by the prometheus-webhook-snmp middleware to define the trap information and delivered object identifiers (OID). Adding the oid or severity labels is only required if you need to change the default values for a particular alert definition. NOTE When you set the oid label, the top-level SNMP trap OID changes, but the sub-OIDs remain defined by the global trapOidPrefix value plus the child OID values .1.1.1 through .1.1.9 . For more information about the MIB definition, see Section 5.4.2, "Overview of the MIB definition" . Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Create a PrometheusRule object that contains the alert rule and an oid label that contains the SNMP trap OID override value: USD oc apply -f - <<EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: creationTimestamp: null labels: prometheus: default role: alert-rules name: prometheus-alarm-rules-snmp namespace: service-telemetry spec: groups: - name: ./openstack.rules rules: - alert: Collectd metrics receive rate is zero expr: rate(sg_total_collectd_msg_received_count[1m]) == 0 labels: oid: 1.3.6.1.4.1.50495.15.1.2.1 severity: critical EOF Additional information For more information about configuring alerts, see Section 5.3, "Alerts in Service Telemetry Framework" . 5.5. Configuring the duration for the TLS certificates To configure the duration of the TLS certificates that you use for the connections with Elasticsearch and AMQ Interconnect in Service Telemetry Framework (STF), modify the ServiceTelemetry object and configure the certificates parameters. 5.5.1. Configuration parameters for the TLS certificates You can configure the duration of the certificate with the following sub-parameters of the certificates parameter: endpointCertDuration The requested duration or lifetime of the endpoint Certificate. Minimum accepted duration is 1 hour. Value must be in units accepted by Go time.ParseDuration https://golang.org/pkg/time/#ParseDuration . The default value is 70080h . caCertDuration The requested duration or lifetime of the CA Certificate. Minimum accepted duration is 1 hour. Value must be in units accepted by Go time.ParseDuration https://golang.org/pkg/time/#ParseDuration . Default value is 70080h . NOTE The default duration of certificates is long, because you usually copy a subset of them in the Red Hat OpenStack Platform deployment when the certificates renew. For more information about the QDR CA Certificate renewal process, see Chapter 6, Renewing the AMQ Interconnect certificate The certificates parameter for Elasticsearch is part of the backends.events.elasticsearch definition and is configured in the ServiceTelemetry object: apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: ... backends: ... events: elasticsearch: enabled: true version: 7.16.1 certificates: endpointCertDuration: 70080h caCertDuration: 70080h ... You can configure the certificates parameter for QDR that is part of the transports.qdr definition in the ServiceTelemetry object: apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: ... transports: ... qdr: enabled: true certificates: endpointCertDuration: 70080h caCertDuration: 70080h ... 5.5.2. Configuring TLS certificates duration To configure the duration of the TLS certificates to use with Service Telemetry Framework (STF), modify the ServiceTelemetry object and configure the certificates parameter. Prerequisites You didn't deploy an instance of Service Telemetry Operator already. NOTE When you create the ServiceTelemetry object, the required certificates and their secrets for STF are also created. For more information about how to modify the certificates and the secrets, see: Chapter 6, Renewing the AMQ Interconnect certificate The following procedure is valid for new STF deployments. Procedure To edit the duration of the TLS certificates, you can set the Elasticsearch endpointCertDuration , for example 26280h for 3 years, and set the QDR caCertDuration , for example 87600h for 10 years. You can use the default value of 8 years for the CA certificate for Elasticsearch and endpoint certificate: + USD oc apply -f - <<EOF apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: backends: events: elasticsearch: enabled: true certificates: endpointCertDuration: 26280h transport: qdr: enabled: true certificates: caCertDuration: 87600h EOF Verification Verify that the expiry date for the certificates is correct: USD oc get secret elasticsearch-es-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -in - -text | grep "Not After" Not After : Mar 9 21:00:16 2026 GMT USD oc get secret default-interconnect-selfsigned -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -in - -text | grep "Not After" Not After : Mar 9 21:00:16 2033 GMT 5.6. High availability With high availability, Service Telemetry Framework (STF) can rapidly recover from failures in its component services. Although Red Hat OpenShift Container Platform restarts a failed pod if nodes are available to schedule the workload, this recovery process might take more than one minute, during which time events and metrics are lost. A high availability configuration includes multiple copies of STF components, which reduces recovery time to approximately 2 seconds. To protect against failure of an Red Hat OpenShift Container Platform node, deploy STF to an Red Hat OpenShift Container Platform cluster with three or more nodes. Warning STF is not yet a fully fault tolerant system. Delivery of metrics and events during the recovery period is not guaranteed. Enabling high availability has the following effects: Three Elasticsearch pods run instead of the default one. The following components run two pods instead of the default one: AMQ Interconnect Alertmanager Prometheus Events Smart Gateway Metrics Smart Gateway Recovery time from a lost pod in any of these services reduces to approximately 2 seconds. 5.6.1. Configuring high availability To configure Service Telemetry Framework (STF) for high availability, add highAvailability.enabled: true to the ServiceTelemetry object in Red Hat OpenShift Container Platform. You can set this parameter at installation time or, if you already deployed STF, complete the following steps: Procedure Log in to Red Hat OpenShift Container Platform. Change to the service-telemetry namespace: USD oc project service-telemetry Use the oc command to edit the ServiceTelemetry object: USD oc edit stf default Add highAvailability.enabled: true to the spec section: apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry ... spec: ... highAvailability: enabled: true Save your changes and close the object. 5.7. Observability Strategy in Service Telemetry Framework Service Telemetry Framework (STF) does not include storage backends and alerting tools. STF uses community operators to deploy Prometheus, Alertmanager, Grafana, and Elasticsearch. STF makes requests to these community operators to create instances of each application configured to work with STF. Instead of having Service Telemetry Operator create custom resource requests, you can use your own deployments of these applications or other compatible applications, and scrape the metrics Smart Gateways for delivery to your own Prometheus-compatible system for telemetry storage. If you set the observabilityStrategy to none , then storage backends will not be deployed so persistent storage will not be required by STF. 5.7.1. Configuring an alternate observability strategy To configure STF to skip the deployment of storage, visualization, and alerting backends, add observabilityStrategy: none to the ServiceTelemetry spec. In this mode, only AMQ Interconnect routers and metrics Smart Gateways are deployed, and you must configure an external Prometheus-compatible system to collect metrics from the STF Smart Gateways. Note Currently, only metrics are supported when you set observabilityStrategy to none . Events Smart Gateways are not deployed. Procedure Create a ServiceTelemetry object with the property observabilityStrategy: none in the spec parameter. The manifest shows results in a default deployment of STF that is suitable for receiving telemetry from a single cloud with all metrics collector types. USD oc apply -f - <<EOF apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: observabilityStrategy: none EOF Delete the left over objects that are managed by community operators USD for o in alertmanager/default prometheus/default elasticsearch/elasticsearch grafana/default; do oc delete USDo; done To verify that all workloads are operating correctly, view the pods and the status of each pod: USD oc get pods NAME READY STATUS RESTARTS AGE default-cloud1-ceil-meter-smartgateway-59c845d65b-gzhcs 3/3 Running 0 132m default-cloud1-coll-meter-smartgateway-75bbd948b9-d5phm 3/3 Running 0 132m default-cloud1-sens-meter-smartgateway-7fdbb57b6d-dh2g9 3/3 Running 0 132m default-interconnect-668d5bbcd6-57b2l 1/1 Running 0 132m interconnect-operator-b8f5bb647-tlp5t 1/1 Running 0 47h service-telemetry-operator-566b9dd695-wkvjq 1/1 Running 0 156m smart-gateway-operator-58d77dcf7-6xsq7 1/1 Running 0 47h Additional resources For more information about configuring additional clouds or to change the set of supported collectors, see Section 4.3.2, "Deploying Smart Gateways" 5.8. Resource usage of Red Hat OpenStack Platform services You can monitor the resource usage of the Red Hat OpenStack Platform (RHOSP) services, such as the APIs and other infrastructure processes, to identify bottlenecks in the overcloud by showing services that run out of compute power. Resource usage monitoring is enabled by default. Additional resources To disable resource usage monitoring, see Section 5.8.1, "Disabling resource usage monitoring of Red Hat OpenStack Platform services" . 5.8.1. Disabling resource usage monitoring of Red Hat OpenStack Platform services To disable the monitoring of RHOSP containerized service resource usage, you must set the CollectdEnableLibpodstats parameter to false . Prerequisites You have created the stf-connectors.yaml file. For more information, see Section 4.1, "Deploying Red Hat OpenStack Platform overcloud for Service Telemetry Framework using director" . You are using the most current version of Red Hat OpenStack Platform (RHOSP) 17.0. Procedure Open the stf-connectors.yaml file and add the CollectdEnableLibpodstats parameter to override the setting in enable-stf.yaml . Ensure that stf-connectors.yaml is called from the openstack overcloud deploy command after enable-stf.yaml : CollectdEnableLibpodstats: false Continue with the overcloud deployment procedure. For more information, see Section 4.1.5, "Deploying the overcloud" . 5.9. Red Hat OpenStack Platform API status and containerized services health You can use the OCI (Open Container Initiative) standard to assess the container health status of each Red Hat OpenStack Platform (RHOSP) service by periodically running a health check script. Most RHOSP services implement a health check that logs issues and returns a binary status. For the RHOSP APIs, the health checks query the root endpoint and determine the health based on the response time. Monitoring of RHOSP container health and API status is enabled by default. Additional resources To disable RHOSP container health and API status monitoring, see Section 5.9.1, "Disabling container health and API status monitoring" . 5.9.1. Disabling container health and API status monitoring To disable RHOSP containerized service health and API status monitoring, you must set the CollectdEnableSensubility parameter to false . Prerequisites You have created the stf-connectors.yaml file in your templates directory. For more information, see Section 4.1, "Deploying Red Hat OpenStack Platform overcloud for Service Telemetry Framework using director" . You are using the most current version of Red Hat OpenStack Platform (RHOSP) 17.0. Procedure Open the stf-connectors.yaml and add the CollectdEnableSensubility parameter to override the setting in enable-stf.yaml . Ensure that stf-connectors.yaml is called from the openstack overcloud deploy command after enable-stf.yaml : CollectdEnableSensubility: false Continue with the overcloud deployment procedure. For more information, see Section 4.1.5, "Deploying the overcloud" . Additional resources For more information about multiple cloud addresses, see Section 4.3, "Configuring multiple clouds" .

Commands:
oc project service-telemetry
oc apply -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: grafana-operator namespace: service-telemetry spec: channel: v4 installPlanApproval: Automatic name: grafana-operator source: community-operators sourceNamespace: openshift-marketplace EOF
oc get csv --selector operators.coreos.com/grafana-operator.service-telemetry NAME DISPLAY VERSION REPLACES PHASE grafana-operator.v4.10.1 Grafana Operator 4.10.1 grafana-operator.v4.10.0 Succeeded
oc edit stf default apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry spec: graphing: enabled: true grafana: ingressEnabled: true baseImage: 'registry.redhat.io/rhel8/grafana:7'
oc get pod -l app=grafana NAME READY STATUS RESTARTS AGE grafana-deployment-7fc7848b56-sbkhv 1/1 Running 0 1m
oc get grafanadatasources NAME AGE default-datasources 20h
oc get route grafana-route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD grafana-route grafana-route-service-telemetry.apps.infra.watch grafana-service 3000 edge None
oc get pod -l "app=grafana" -ojsonpath='{.items[0].spec.containers[0].image}' docker.io/grafana/grafana:7.3.10
oc patch stf/default --type merge -p '{"spec":{"graphing":{"grafana":{"baseImage":"docker.io/grafana/grafana:8.1.5"}}}}'
oc get pod -l "app=grafana" NAME READY STATUS RESTARTS AGE grafana-deployment-fb9799b58-j2hj2 1/1 Running 0 10s
oc get pod -l "app=grafana" -ojsonpath='{.items[0].spec.containers[0].image}' docker.io/grafana/grafana:8.1.0
oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-dashboard.yaml grafanadashboard.integreatly.org/rhos-dashboard-1 created
oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-cloud-dashboard.yaml grafanadashboard.integreatly.org/rhos-cloud-dashboard-1 created
oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-cloudevents-dashboard.yaml grafanadashboard.integreatly.org/rhos-cloudevents-dashboard created
oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/virtual-machine-view.yaml grafanadashboard.integreatly.org/virtual-machine-view-1 configured
oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/memcached-dashboard.yaml grafanadashboard.integreatly.org/memcached-dashboard-1 created
oc get grafanadashboards NAME AGE memcached-dashboard-1 7s rhos-cloud-dashboard-1 23s rhos-cloudevents-dashboard 18s rhos-dashboard-1 29s virtual-machine-view-1 13s
oc get route grafana-route -ojsonpath='{.spec.host}' grafana-route-service-telemetry.apps.infra.watch
oc project service-telemetry
oc get stf default -o jsonpath="{.spec.graphing.grafana['adminUser','adminPassword']}"
oc edit stf default
oc get po -l app=grafana -w
oc project service-telemetry
oc edit stf default
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: backends: metrics: prometheus: enabled: true storage: strategy: persistent retention: 7d
oc get po -l app.kubernetes.io/name=prometheus -w
oc describe po prometheus-default-0 | grep retention.time --storage.tsdb.retention.time=24h
oc project service-telemetry
oc apply -f - <<EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: creationTimestamp: null labels: prometheus: default role: alert-rules name: prometheus-alarm-rules namespace: service-telemetry spec: groups: - name: ./openstack.rules rules: - alert: Collectd metrics receive rate is zero expr: rate(sg_total_collectd_msg_received_count[1m]) == 0 EOF
curl -k --user "internal:USD(oc get secret default-prometheus-htpasswd -ogo-template='{{ .data.password | base64decode }}')" https://USD(oc get route default-prometheus-proxy -ogo-template='{{ .spec.host }}')/api/v1/rules {"status":"success","data":{"groups":[{"name":"./openstack.rules","file":"/etc/prometheus/rules/prometheus-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml","rules":[{"state":"inactive","name":"Collectd metrics receive count is zero","query":"rate(sg_total_collectd_msg_received_count[1m]) == 0","duration":0,"labels":{},"annotations":{},"alerts":[],"health":"ok","evaluationTime":0.00034627,"lastEvaluation":"2021-12-07T17:23:22.160448028Z","type":"alerting"}],"interval":30,"evaluationTime":0.000353787,"lastEvaluation":"2021-12-07T17:23:22.160444017Z"}]}}
oc edit prometheusrules prometheus-alarm-rules
alertmanager.yaml: |- global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
oc project service-telemetry
oc edit stf default
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: backends: metrics: prometheus: enabled: true alertmanagerConfigManifest: | apiVersion: v1 kind: Secret metadata: name: 'alertmanager-default' namespace: 'service-telemetry' type: Opaque stringData: alertmanager.yaml: |- global: resolve_timeout: 10m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}' global: resolve_timeout: 10m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
oc exec -it prometheus-default-0 -c prometheus -- sh -c "wget --header \"Authorization: Bearer \USD(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -" {"status":"success","data":{"configYAML":"...",...}}
oc delete pod curl pod "curl" deleted
alertmanager.yaml: |- global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
oc project service-telemetry
cat > alertmanager.yaml <<EOF global: resolve_timeout: 10m slack_api_url: <slack_api_url> receivers: - name: slack slack_configs: - channel: #stf-alerts title: |- text: >- route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack' EOF
CONFIG_MANIFEST=USD(oc create secret --dry-run=client generic alertmanager-default --from-file=alertmanager.yaml -o json) oc patch stf default --type=merge -p '{"spec":{"alertmanagerConfigManifest":'"USDCONFIG_MANIFEST"'}}'
oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}' global: resolve_timeout: 10m slack_api_url: <slack_api_url> receivers: - name: slack slack_configs: - channel: #stf-alerts title: |- text: >- route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack'
oc exec -it prometheus-default-0 -c prometheus -- /bin/sh -c "wget --header \"Authorization: Bearer \USD(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -" {"status":"success","data":{"configYAML":"...",...}}
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: alerting: alertmanager: receivers: snmpTraps: alertOidLabel: oid community: public enabled: true port: 162 retries: 5 target: 192.168.25.254 timeout: 1 trapDefaultOid: 1.3.6.1.4.1.50495.15.1.2.1 trapDefaultSeverity: "" trapOidPrefix: 1.3.6.1.4.1.50495.15
SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.50495.15.1.2.1 SNMPv2-SMI::enterprises.50495.15.1.1.1 = STRING: "TEST ALERT FROM PROMETHEUS PLEASE ACKNOWLEDGE" SNMPv2-SMI::enterprises.50495.15.1.1.2 = STRING: "firing" SNMPv2-SMI::enterprises.50495.15.1.1.3 = STRING: "warning" SNMPv2-SMI::enterprises.50495.15.1.1.4 = "" SNMPv2-SMI::enterprises.50495.15.1.1.5 = "" SNMPv2-SMI::enterprises.50495.15.1.1.6 = STRING: "TEST ALERT FROM " SNMPv2-SMI::enterprises.50495.15.1.1.7 = STRING: "{\"cluster\": \"TEST\", \"container\": \"sg-core\", \"endpoint\": \"prom-https\", \"prometheus\": \"service-telemetry/default\", \"service\": \"default-cloud1-coll-meter\", \"source\": \"SG\"}" SNMPv2-SMI::enterprises.50495.15.1.1.8 = Timeticks: (1676476389) 194 days, 0:52:43.89 SNMPv2-SMI::enterprises.50495.15.1.1.9 = STRING: "{\"status\": \"firing\", \"labels\": {\"cluster\": \"TEST\", \"container\": \"sg-core\", \"endpoint\": \"prom-https\", \"prometheus\": \"service-telemetry/default\", \"service\": \"default-cloud1-coll-meter\", \"source\": \"SG\"}, \"annotations\": {\"action\": \"TESTING PLEASE ACKNOWLEDGE, NO FURTHER ACTION REQUIRED ONLY A TEST\"}, \"startsAt\": \"2023-02-15T15:53:09.109Z\", \"endsAt\": \"0001-01-01T00:00:00Z\", \"generatorURL\": \"http://prometheus-default-0:9090/graph?g0.expr=sg_total_collectd_msg_received_count+%3E+1&g0.tab=1\", \"fingerprint\": \"feefeb77c577a02f\"}"
oc project service-telemetry
oc edit stf default
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry spec: alerting: alertmanager: receivers: snmpTraps: enabled: true target: 10.10.10.10
oc project service-telemetry
oc apply -f - <<EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: creationTimestamp: null labels: prometheus: default role: alert-rules name: prometheus-alarm-rules-snmp namespace: service-telemetry spec: groups: - name: ./openstack.rules rules: - alert: Collectd metrics receive rate is zero expr: rate(sg_total_collectd_msg_received_count[1m]) == 0 labels: oid: 1.3.6.1.4.1.50495.15.1.2.1 severity: critical EOF
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: backends: events: elasticsearch: enabled: true version: 7.16.1 certificates: endpointCertDuration: 70080h caCertDuration: 70080h
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: transports: qdr: enabled: true certificates: endpointCertDuration: 70080h caCertDuration: 70080h
oc apply -f - <<EOF apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: backends: events: elasticsearch: enabled: true certificates: endpointCertDuration: 26280h transport: qdr: enabled: true certificates: caCertDuration: 87600h EOF
oc get secret elasticsearch-es-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -in - -text | grep "Not After" Not After : Mar 9 21:00:16 2026 GMT oc get secret default-interconnect-selfsigned -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -in - -text | grep "Not After" Not After : Mar 9 21:00:16 2033 GMT
oc project service-telemetry
oc edit stf default
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry spec: highAvailability: enabled: true
oc apply -f - <<EOF apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: observabilityStrategy: none EOF
for o in alertmanager/default prometheus/default elasticsearch/elasticsearch grafana/default; do oc delete USDo; done
oc get pods NAME READY STATUS RESTARTS AGE default-cloud1-ceil-meter-smartgateway-59c845d65b-gzhcs 3/3 Running 0 132m default-cloud1-coll-meter-smartgateway-75bbd948b9-d5phm 3/3 Running 0 132m default-cloud1-sens-meter-smartgateway-7fdbb57b6d-dh2g9 3/3 Running 0 132m default-interconnect-668d5bbcd6-57b2l 1/1 Running 0 132m interconnect-operator-b8f5bb647-tlp5t 1/1 Running 0 47h service-telemetry-operator-566b9dd695-wkvjq 1/1 Running 0 156m smart-gateway-operator-58d77dcf7-6xsq7 1/1 Running 0 47h
CollectdEnableLibpodstats: false
CollectdEnableSensubility: false