Chapter 4. Configuring the Cluster Observability Operator to monitor a service You can monitor metrics for a service by configuring monitoring stacks managed by the Cluster Observability Operator (COO). To test monitoring a service, follow these steps: Deploy a sample service that defines a service endpoint. Create a ServiceMonitor object that specifies how the service is to be monitored by the COO. Create a MonitoringStack object to discover the ServiceMonitor object. 4.1. Deploying a sample service for Cluster Observability Operator This configuration deploys a sample service named prometheus-coo-example-app in the user-defined ns1-coo project. The service exposes the custom version metric. Prerequisites You have access to the cluster as a user with the cluster-admin cluster role or as a user with administrative permissions for the namespace. Procedure Create a YAML file named prometheus-coo-example-app.yaml that contains the following configuration details for a namespace, deployment, and service: apiVersion: v1 kind: Namespace metadata: name: ns1-coo --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: prometheus-coo-example-app name: prometheus-coo-example-app namespace: ns1-coo spec: replicas: 1 selector: matchLabels: app: prometheus-coo-example-app template: metadata: labels: app: prometheus-coo-example-app spec: containers: - image: ghcr.io/rhobs/prometheus-example-app:0.4.2 imagePullPolicy: IfNotPresent name: prometheus-coo-example-app --- apiVersion: v1 kind: Service metadata: labels: app: prometheus-coo-example-app name: prometheus-coo-example-app namespace: ns1-coo spec: ports: - port: 8080 protocol: TCP targetPort: 8080 name: web selector: app: prometheus-coo-example-app type: ClusterIP Save the file. Apply the configuration to the cluster by running the following command: USD oc apply -f prometheus-coo-example-app.yaml Verify that the pod is running by running the following command and observing the output: USD oc -n ns1-coo get pod Example output NAME READY STATUS RESTARTS AGE prometheus-coo-example-app-0927545cb7-anskj 1/1 Running 0 81m 4.2. Specifying how a service is monitored by Cluster Observability Operator To use the metrics exposed by the sample service you created in the "Deploying a sample service for Cluster Observability Operator" section, you must configure monitoring components to scrape metrics from the /metrics endpoint. You can create this configuration by using a ServiceMonitor object that specifies how the service is to be monitored, or a PodMonitor object that specifies how a pod is to be monitored. The ServiceMonitor object requires a Service object. The PodMonitor object does not, which enables the MonitoringStack object to scrape metrics directly from the metrics endpoint exposed by a pod. This procedure shows how to create a ServiceMonitor object for a sample service named prometheus-coo-example-app in the ns1-coo namespace. Prerequisites You have access to the cluster as a user with the cluster-admin cluster role or as a user with administrative permissions for the namespace. You have installed the Cluster Observability Operator. You have deployed the prometheus-coo-example-app sample service in the ns1-coo namespace. Note The prometheus-coo-example-app sample service does not support TLS authentication. Procedure Create a YAML file named example-coo-app-service-monitor.yaml that contains the following ServiceMonitor object configuration details: apiVersion: monitoring.rhobs/v1 kind: ServiceMonitor metadata: labels: k8s-app: prometheus-coo-example-monitor name: prometheus-coo-example-monitor namespace: ns1-coo spec: endpoints: - interval: 30s port: web scheme: http selector: matchLabels: app: prometheus-coo-example-app This configuration defines a ServiceMonitor object that the MonitoringStack object will reference to scrape the metrics data exposed by the prometheus-coo-example-app sample service. Apply the configuration to the cluster by running the following command: USD oc apply -f example-coo-app-service-monitor.yaml Verify that the ServiceMonitor resource is created by running the following command and observing the output: USD oc -n ns1-coo get servicemonitors.monitoring.rhobs Example output NAME AGE prometheus-coo-example-monitor 81m 4.3. Creating a MonitoringStack object for the Cluster Observability Operator To scrape the metrics data exposed by the target prometheus-coo-example-app service, create a MonitoringStack object that references the ServiceMonitor object you created in the "Specifying how a service is monitored for Cluster Observability Operator" section. This MonitoringStack object can then discover the service and scrape the exposed metrics data from it. Prerequisites You have access to the cluster as a user with the cluster-admin cluster role or as a user with administrative permissions for the namespace. You have installed the Cluster Observability Operator. You have deployed the prometheus-coo-example-app sample service in the ns1-coo namespace. You have created a ServiceMonitor object named prometheus-coo-example-monitor in the ns1-coo namespace. Procedure Create a YAML file for the MonitoringStack object configuration. For this example, name the file example-coo-monitoring-stack.yaml . Add the following MonitoringStack object configuration details: Example MonitoringStack object apiVersion: monitoring.rhobs/v1alpha1 kind: MonitoringStack metadata: name: example-coo-monitoring-stack namespace: ns1-coo spec: logLevel: debug retention: 1d resourceSelector: matchLabels: k8s-app: prometheus-coo-example-monitor Apply the MonitoringStack object by running the following command: USD oc apply -f example-coo-monitoring-stack.yaml Verify that the MonitoringStack object is available by running the following command and inspecting the output: USD oc -n ns1-coo get monitoringstack Example output NAME AGE example-coo-monitoring-stack 81m Run the following comand to retrieve information about the active targets from Prometheus and filter the output to list only targets labeled with app=prometheus-coo-example-app . This verifies which targets are discovered and actively monitored by Prometheus with this specific label. USD oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")' Example output { "__address__": "10.129.2.25:8080", "__meta_kubernetes_endpoint_address_target_kind": "Pod", "__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "__meta_kubernetes_endpoint_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz", "__meta_kubernetes_endpoint_port_name": "web", "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "true", "__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2024-11-05T11:24:09Z", "__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true", "__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app", "__meta_kubernetes_endpoints_labelpresent_app": "true", "__meta_kubernetes_endpoints_name": "prometheus-coo-example-app", "__meta_kubernetes_namespace": "ns1-coo", "__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.129.2.25/23\"],\"mac_address\":\"0a:58:0a:81:02:19\",\"gateway_ips\":[\"10.129.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.129.2.1\"}],\"ip_address\":\"10.129.2.25/23\",\"gateway_ip\":\"10.129.2.1\",\"role\":\"primary\"}}", "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.129.2.25\"\n ],\n \"mac\": \"0a:58:0a:81:02:19\",\n \"default\": true,\n \"dns\": {}\n}]", "__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2", "__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default", "__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true", "__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true", "__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true", "__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true", "__meta_kubernetes_pod_controller_kind": "ReplicaSet", "__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-5d8cd498c7", "__meta_kubernetes_pod_host_ip": "10.0.128.2", "__meta_kubernetes_pod_ip": "10.129.2.25", "__meta_kubernetes_pod_label_app": "prometheus-coo-example-app", "__meta_kubernetes_pod_label_pod_template_hash": "5d8cd498c7", "__meta_kubernetes_pod_labelpresent_app": "true", "__meta_kubernetes_pod_labelpresent_pod_template_hash": "true", "__meta_kubernetes_pod_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "__meta_kubernetes_pod_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz", "__meta_kubernetes_pod_phase": "Running", "__meta_kubernetes_pod_ready": "true", "__meta_kubernetes_pod_uid": "054c11b6-9a76-4827-a860-47f3a4596871", "__meta_kubernetes_service_label_app": "prometheus-coo-example-app", "__meta_kubernetes_service_labelpresent_app": "true", "__meta_kubernetes_service_name": "prometheus-coo-example-app", "__metrics_path__": "/metrics", "__scheme__": "http", "__scrape_interval__": "30s", "__scrape_timeout__": "10s", "job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0" } Note The above example uses jq command-line JSON processor to format the output for convenience. 4.4. Validating the monitoring stack To validate that the monitoring stack is working correctly, access the example service and then view the gathered metrics. Prerequisites You have access to the cluster as a user with the cluster-admin cluster role or as a user with administrative permissions for the namespace. You have installed the Cluster Observability Operator. You have deployed the prometheus-coo-example-app sample service in the ns1-coo namespace. You have created a ServiceMonitor object named prometheus-coo-example-monitor in the ns1-coo namespace. You have created a MonitoringStack object named example-coo-monitoring-stack in the ns1-coo namespace. Procedure Create a route to expose the example prometheus-coo-example-app service. From your terminal, run the command: USD oc expose svc prometheus-coo-example-app -n ns1-coo Access the route from your browser, or command line, to generate metrics. Execute a query on the Prometheus pod to return the total HTTP requests metric: USD oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total' Example output (formatted using jq for convenience) { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "http_requests_total", "code": "200", "endpoint": "web", "instance": "10.129.2.25:8080", "job": "prometheus-coo-example-app", "method": "get", "namespace": "ns1-coo", "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "service": "prometheus-coo-example-app" }, "value": [ 1730807483.632, "3" ] }, { "metric": { "__name__": "http_requests_total", "code": "404", "endpoint": "web", "instance": "10.129.2.25:8080", "job": "prometheus-coo-example-app", "method": "get", "namespace": "ns1-coo", "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "service": "prometheus-coo-example-app" }, "value": [ 1730807483.632, "0" ] } ] } }

Commands:
apiVersion: v1 kind: Namespace metadata: name: ns1-coo --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: prometheus-coo-example-app name: prometheus-coo-example-app namespace: ns1-coo spec: replicas: 1 selector: matchLabels: app: prometheus-coo-example-app template: metadata: labels: app: prometheus-coo-example-app spec: containers: - image: ghcr.io/rhobs/prometheus-example-app:0.4.2 imagePullPolicy: IfNotPresent name: prometheus-coo-example-app --- apiVersion: v1 kind: Service metadata: labels: app: prometheus-coo-example-app name: prometheus-coo-example-app namespace: ns1-coo spec: ports: - port: 8080 protocol: TCP targetPort: 8080 name: web selector: app: prometheus-coo-example-app type: ClusterIP
oc apply -f prometheus-coo-example-app.yaml
oc -n ns1-coo get pod
NAME READY STATUS RESTARTS AGE prometheus-coo-example-app-0927545cb7-anskj 1/1 Running 0 81m
apiVersion: monitoring.rhobs/v1 kind: ServiceMonitor metadata: labels: k8s-app: prometheus-coo-example-monitor name: prometheus-coo-example-monitor namespace: ns1-coo spec: endpoints: - interval: 30s port: web scheme: http selector: matchLabels: app: prometheus-coo-example-app
oc apply -f example-coo-app-service-monitor.yaml
oc -n ns1-coo get servicemonitors.monitoring.rhobs
NAME AGE prometheus-coo-example-monitor 81m
apiVersion: monitoring.rhobs/v1alpha1 kind: MonitoringStack metadata: name: example-coo-monitoring-stack namespace: ns1-coo spec: logLevel: debug retention: 1d resourceSelector: matchLabels: k8s-app: prometheus-coo-example-monitor
oc apply -f example-coo-monitoring-stack.yaml
oc -n ns1-coo get monitoringstack
NAME AGE example-coo-monitoring-stack 81m
oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")'
{ "__address__": "10.129.2.25:8080", "__meta_kubernetes_endpoint_address_target_kind": "Pod", "__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "__meta_kubernetes_endpoint_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz", "__meta_kubernetes_endpoint_port_name": "web", "__meta_kubernetes_endpoint_port_protocol": "TCP", "__meta_kubernetes_endpoint_ready": "true", "__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2024-11-05T11:24:09Z", "__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true", "__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app", "__meta_kubernetes_endpoints_labelpresent_app": "true", "__meta_kubernetes_endpoints_name": "prometheus-coo-example-app", "__meta_kubernetes_namespace": "ns1-coo", "__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.129.2.25/23\"],\"mac_address\":\"0a:58:0a:81:02:19\",\"gateway_ips\":[\"10.129.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.129.2.1\"}],\"ip_address\":\"10.129.2.25/23\",\"gateway_ip\":\"10.129.2.1\",\"role\":\"primary\"}}", "__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.129.2.25\"\n ],\n \"mac\": \"0a:58:0a:81:02:19\",\n \"default\": true,\n \"dns\": {}\n}]", "__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2", "__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default", "__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true", "__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true", "__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true", "__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true", "__meta_kubernetes_pod_controller_kind": "ReplicaSet", "__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-5d8cd498c7", "__meta_kubernetes_pod_host_ip": "10.0.128.2", "__meta_kubernetes_pod_ip": "10.129.2.25", "__meta_kubernetes_pod_label_app": "prometheus-coo-example-app", "__meta_kubernetes_pod_label_pod_template_hash": "5d8cd498c7", "__meta_kubernetes_pod_labelpresent_app": "true", "__meta_kubernetes_pod_labelpresent_pod_template_hash": "true", "__meta_kubernetes_pod_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "__meta_kubernetes_pod_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz", "__meta_kubernetes_pod_phase": "Running", "__meta_kubernetes_pod_ready": "true", "__meta_kubernetes_pod_uid": "054c11b6-9a76-4827-a860-47f3a4596871", "__meta_kubernetes_service_label_app": "prometheus-coo-example-app", "__meta_kubernetes_service_labelpresent_app": "true", "__meta_kubernetes_service_name": "prometheus-coo-example-app", "__metrics_path__": "/metrics", "__scheme__": "http", "__scrape_interval__": "30s", "__scrape_timeout__": "10s", "job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0" }
oc expose svc prometheus-coo-example-app -n ns1-coo
oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total'
{ "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "http_requests_total", "code": "200", "endpoint": "web", "instance": "10.129.2.25:8080", "job": "prometheus-coo-example-app", "method": "get", "namespace": "ns1-coo", "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "service": "prometheus-coo-example-app" }, "value": [ 1730807483.632, "3" ] }, { "metric": { "__name__": "http_requests_total", "code": "404", "endpoint": "web", "instance": "10.129.2.25:8080", "job": "prometheus-coo-example-app", "method": "get", "namespace": "ns1-coo", "pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj", "service": "prometheus-coo-example-app" }, "value": [ 1730807483.632, "0" ] } ] } }