Chapter 3. Enabling 3scale API Management monitoring You can enable the Red Hat 3scale API Management monitoring using Prometheus and Grafana operators. Prometheus is an open-source monitoring and alerting tool designed to collect and analyze metrics from applications and infrastructure in real time. Grafana is an open-source analytics and visualization platform used to display and monitor real-time data from various sources in customizable dashboards. Important With the deprecation of Grafana 4 on OpenShift 4.16 and later, the instructions are split into two sections: For OpenShift 4.15 and earlier. For OpenShift 4.16 and later. Grafana 5 is available on OpenShift 4.14 or later and supported by the 3scale 2.15 operator; using Grafana 5 is recommended when possible. Red Hat support for Prometheus and Grafana is limited to the configuration recommendations provided in Red Hat product documentation. The 3scale operator creates monitoring resources, but does not prevent modification of those resources. You must install the 3scale operator and Prometheus operator in the same namespace or use cluster-wide operators. The steps for migrating from Grafana 4 to Grafana 5 are also covered. Prerequisites The 3scale operator is installed . The Prometheus operator is installed from the OperatorHub . You can use the Prometheus operator for creating and managing Prometheus instances. It provides the Prometheus custom resource definition (CRD) required by 3scale monitoring. The following Prometheus operator version is compatible with 3scale depending on your version of OpenShift Container Platform (OCP): Latest Prometheus community operator with OCP 4.15 or earlier. Latest Prometheus community operator with OCP 4.16 or later. The Grafana operator is installed from the OperatorHub . You can use the Grafana operator for creating and managing Grafana instances. It provides the GrafanaDashboard CRD required by 3scale monitoring. The following Grafana operator versions are compatible with 3scale depending on your version of OCP: Grafana community operator 4 with OCP 4.15 or earlier. Grafana community operator 5 with OCP 4.16 or later. Important If your cluster is exposed on the Internet, make sure to protect the Prometheus and Grafana services. This section describes how to enable monitoring of a 3scale instance, so that you can view the Grafana dashboards. Configuring Grafana and Prometheus operators v4 up to OCP 4.15 Configuring Grafana to monitor 3scale API Management Viewing metrics for 3scale API Management 3scale API Management system metrics exposed to Prometheus 3.1. Configuring Grafana 4 and Prometheus to OCP 4.15 Configure Grafana and Prometheus for OpenShift Container Platform (OCP) versions up to 4.15. This guide walks through setting up these tools to monitor your 3scale environment, ensuring you get real-time insights and data visualization. Procedure Install Grafana 4 community operator from the OperatorHub . Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials. Select the project from the Project list where you want to install the Grafana community operator. Important Install the Grafana community operator in the same project where you installed the 3scale operator. Navigate to Operators > OperatorHub . Search for "grafana" and click Grafana Operator . Click Install on the Grafana Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Grafana operator subscription: Click A specific namespace on the cluster and choose the project where you want to install the Grafana community operator. Click Subscribe . Click Approve . Install latest Prometheus community operator from the OperatorHub . Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials. Select the project from the Project list where you want to install the Prometheus community operator. Important Install the Prometheus community operator in the same project where you installed the 3scale operator. Navigate to Operators > OperatorHub . Search for "prometheus" and click Prometheus Operator . Click Install on the Prometheus Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Prometheus operator subscription: Click A specific namespace on the cluster and choose the project where you want to install the Prometheus operator. Click Subscribe . Click Approve . Enable monitoring by setting the spec.monitoring.enabled parameter of your 3scale deployment YAML to true . For example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: ... spec: monitoring: enabled: true ... Log in to your OpenShift cluster. You must log in as a user with an edit cluster role in the OpenShift project of the 3scale, for example, cluster-admin . USD oc login Switch to your 3scale project. USD oc project <project_name> Create a new service account for Prometheus: USD oc create serviceaccount prometheus-monitoring Create a ClusterRoleBinding to give Prometheus ServiceAccount the Role-based access control (RBAC) permissions required to scrape metrics. Update the ServiceAccount namespace before creating the ClusterRoleBinding: USD oc adm policy add-cluster-role-to-user cluster-monitoring-view -z prometheus-monitoring -n "<3scale_namespace> Create a token for the prometheus-monitoring ServiceAccount: USD oc create token prometheus-monitoring Prometheus will lose access to the required resource when this token expires. Add the --duration X[s|m|h] to specify how long the token is valid for. Update the file 3scale-scrape-configs.yaml bearer_token field with the token generated you generated. Create additional-scrape-config secret: USD oc create secret generic additional-scrape-configs --from-file=3scale-scrape-configs.yaml=./3scale-scrape-configs.yaml Deploy Prometheus. In Prometheus.yaml file, fill the spec.externalUrl field with the external URL. The URL template should be: spec: ... externalUrl: https://prometheus.<namespace-name>.apps.<cluster-domain> Deploy Prometheus server: USD oc apply -f prometheus.yaml Create the Prometheus route: USD oc expose service prometheus-operated --hostname prometheus.<namespace-name>.apps.<cluster-name> Deploy Grafana datasource: USD oc apply -f datasource-v4.yaml Deploy Grafana USD oc apply -f grafana-v4.yaml Additional resources Prometheus documentation 3.2. Configuring Grafana and Prometheus for OCP 4.16 Configure Grafana and Prometheus for OpenShift Container Platform (OCP) 4.16. This guide covers the steps to set up monitoring and data visualization, optimized for OCP 4.16. Procedure Install Grafana 5 community operator from the OperatorHub . Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials. Select the project from the Project list where you want to install the Grafana community operator. Navigate to Operators > OperatorHub . Search for "grafana" and click Grafana Operator . Click Install on the Grafana Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Grafana operator subscription: Click A specific namespace on the cluster and choose the project where you want to install the Grafana community operator. Click Subscribe . Click Approve . Install the latest Prometheus community operator from the OperatorHub . Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials. Select the project from the Project list where you want to install the Prometheus community operator. Important Install the Prometheus community operator in the same project where you installed the 3scale operator. Navigate to Operators > OperatorHub . Search for "prometheus" and click Prometheus Operator . Click Install on the Prometheus Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Prometheus operator subscription: Click A specific namespace on the cluster and choose the project where you want to install the Prometheus community operator. Click Subscribe . Click Approve . Enable monitoring by setting the spec.monitoring.enabled parameter of your 3scale deployment YAML to true . For example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: ... spec: monitoring: enabled: true ... Log in to your OpenShift cluster. You must log in as a user with an edit cluster role in the OpenShift project of the 3scale, for example, cluster-admin . USD oc login Switch to your 3scale project. USD oc project <project_name> Create a new service account for Prometheus: USD oc create serviceaccount prometheus-monitoring Create a ClusterRoleBinding to give Prometheus ServiceAccount the Role-based access control (RBAC) permissions required to scrape metrics. Update the ServiceAccount namespace before creating the ClusterRoleBinding: USD oc adm policy add-cluster-role-to-user cluster-monitoring-view -z prometheus-monitoring -n "<3scale_namespace>" Create a token for the prometheus-monitoring ServiceAccount: USD oc create token prometheus-monitoring Prometheus will lose access to the required resource when this token expires. Add the --duration X[s|m|h] to specify how long the token is valid for. Update the file 3scale-scrape-configs.yaml bearer_token field with the token generated you generated. Create additional-scrape-config secret: USD oc create secret generic additional-scrape-configs --from-file=3scale-scrape-configs.yaml=./3scale-scrape-configs.yaml Deploy Prometheus. In Prometheus.yaml file, fill the spec.externalUrl field with the external URL. The URL template should be: spec: ... externalUrl: https://prometheus.<namespace-name>.apps.<cluster-domain> Deploy Prometheus server: USD oc apply -f prometheus.yaml Create the Prometheus route: USD oc expose service prometheus-operated --hostname prometheus.<namespace-name>.apps.<cluster-name> Deploy Grafana datasource: USD oc apply -f datasource-v5.yaml Deploy Grafana: USD oc apply -f grafana-v5.yaml Expose grafana route: USD oc expose service example-grafana-service Additional resources Grafana documentation getting started Grafana documentation operator flags 3.3. Migrating Grafana 4 to Grafana 5 on OpenShift Container Platform 4.16 This step-by-step guide explains how to migrate from Grafana 4 to Grafana 5 on OpenShift Container Platform (OCP) 4.16 or later. The process covers removing Grafana 4, setting up Grafana 5, and ensuring all necessary components are updated. Important Make sure to back up any important data and dashboards before starting the migration. 3.3.1. Removing Grafana 4 Procedure Remove the Grafana 4 custom resource (CR): Important Removing the Grafana 4 CR deletes the Grafana 4 application along with the service and routes. USD oc delete grafana <grafana-cr-name> -n <namespace> Verify route removal: USD oc get route -n <namespace> Delete the route manually if it still exists: USD oc delete route <grafana-route-name> -n <namespace> Remove the Grafana 4 datasource CR: USD oc delete grafanadatasource <datasource-cr-name> -n <namespace> Remove the Grafana 4 operator: USD oc delete subscription <grafana-operator-subscription> -n <namespace> USD oc delete clusterserviceversion <grafana-operator-csv-name> -n <namespace> 3.3.2. Installing Grafana 5 Procedure Install Grafana 5: Follow the procedure to Configure Grafana and Prometheus for OCP 4.16 Restart the Prometheus instance to apply the new configuration: 3.3.3. Migrating custom dashboards to Grafana 5 CRDs Procedure Migrate your custom dashboards to Grafana 5 CRDs. Convert the exported JSON dashboards to Custom Resource Definitions (CRDs) for Grafana v5. Example YAML for a GrafanaDashboard CR: apiVersion: integreatly.org/v1alpha1 kind: GrafanaDashboard metadata: name: example-dashboard namespace: <namespace> spec: json: | { "dashboard": { ... } } Apply the dashboard CRD: USD oc apply -f grafana-dashboard.yaml 3.3.4. Optional: Removing Grafana 4 CRDs Procedure Delete Grafana 4 dashboards CRDs: USD oc delete crd grafanadashboards.integreatly.org Important If you choose not to remove the CRDs, the 3scale operator continues to reconcile Grafana 4 dashboards, but they do not have any impact on the monitoring stack. 3.3.5. Restarting Grafana operator and Grafana deployment Procedure Restart the 3scale operator: USD oc rollout restart deployment/3scale-operator -n <namespace> Restart Grafana operator and Grafana deployment: USD oc rollout restart deployment/grafana-operator -n <namespace> USD oc rollout restart deployment/<grafana-deployment-name> -n <namespace> 3.4. Viewing metrics for 3scale API Management After configuring 3scale, Prometheus, and Grafana you can view the metrics described in this section. Procedure Log into the Grafana console. Check that you can view metrics for the following: Kubernetes resources at pod and namespace level where 3scale is installed APIcast Staging APIcast Production Backend worker Backend listener System Zync 3.5. 3scale API Management system metrics exposed to Prometheus You can configure the following ports to use 3scale system pods with Prometheus endpoints to expose metrics. Table 3.1. 3scale system ports system-app Port system-developer 9394 system-master 9395 system-provider 9396 system-sidekiq Port system-sidekiq 9394 The endpoints are only accessible internally using: For example: Additional resources Exposing 3scale API Management APIcast Metrics to Prometheus Prometheus security Grafana permissions Grafana security Deployment configuration options for 3scale API Management on OpenShift using the operator 3scale PrometheusRules

Commands:
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: spec: monitoring: enabled: true
oc login
oc project <project_name>
oc create serviceaccount prometheus-monitoring
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z prometheus-monitoring -n "<3scale_namespace>
oc create token prometheus-monitoring
oc create secret generic additional-scrape-configs --from-file=3scale-scrape-configs.yaml=./3scale-scrape-configs.yaml
spec: externalUrl: https://prometheus.<namespace-name>.apps.<cluster-domain>
oc apply -f prometheus.yaml
oc expose service prometheus-operated --hostname prometheus.<namespace-name>.apps.<cluster-name>
oc apply -f datasource-v4.yaml
oc apply -f grafana-v4.yaml
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: spec: monitoring: enabled: true
oc login
oc project <project_name>
oc create serviceaccount prometheus-monitoring
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z prometheus-monitoring -n "<3scale_namespace>"
oc create token prometheus-monitoring
oc create secret generic additional-scrape-configs --from-file=3scale-scrape-configs.yaml=./3scale-scrape-configs.yaml
spec: externalUrl: https://prometheus.<namespace-name>.apps.<cluster-domain>
oc apply -f prometheus.yaml
oc expose service prometheus-operated --hostname prometheus.<namespace-name>.apps.<cluster-name>
oc apply -f datasource-v5.yaml
oc apply -f grafana-v5.yaml
oc expose service example-grafana-service
oc delete grafana <grafana-cr-name> -n <namespace>
oc get route -n <namespace>
oc delete route <grafana-route-name> -n <namespace>
oc delete grafanadatasource <datasource-cr-name> -n <namespace>
oc delete subscription <grafana-operator-subscription> -n <namespace> oc delete clusterserviceversion <grafana-operator-csv-name> -n <namespace>
oc rollout restart statefulset prometheus-k8s -n <namespace>
apiVersion: integreatly.org/v1alpha1 kind: GrafanaDashboard metadata: name: example-dashboard namespace: <namespace> spec: json: | { "dashboard": { ... } }
oc apply -f grafana-dashboard.yaml
oc delete crd grafanadashboards.integreatly.org
oc rollout restart deployment/3scale-operator -n <namespace>
oc rollout restart deployment/grafana-operator -n <namespace> oc rollout restart deployment/<grafana-deployment-name> -n <namespace>
http://USD{service}:USD{port}/metrics
http://system-developer:9394/metrics