Chapter 5. Accessing metrics 5.1. Accessing metrics as an administrator You can access metrics to monitor the performance of cluster components and your workloads. Additional resources Understanding metrics 5.1.1. Viewing a list of available metrics As a cluster administrator or as a user with view permissions for all projects, you can view a list of metrics available in a cluster and output the list in JSON format. Prerequisites You are a cluster administrator, or you have access to the cluster as a user with the cluster-monitoring-view cluster role. You have installed the OpenShift Container Platform CLI ( oc ). You have obtained the OpenShift Container Platform API route for Thanos Querier. You are able to get a bearer token by using the oc whoami -t command. Important You can only use bearer token authentication to access the Thanos Querier API route. Procedure If you have not obtained the OpenShift Container Platform API route for Thanos Querier, run the following command: USD oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}' Retrieve a list of metrics in JSON format from the Thanos Querier API route by running the following command. This command uses oc to authenticate with a bearer token. USD curl -k -H "Authorization: Bearer USD(oc whoami -t)" https://<thanos_querier_route>/api/v1/metadata 1 1 Replace <thanos_querier_route> with the OpenShift Container Platform API route for Thanos Querier. 5.1.2. Querying metrics for all projects with the OpenShift Container Platform web console You can use the OpenShift Container Platform metrics query browser to run Prometheus Query Language (PromQL) queries to examine metrics visualized on a plot. This functionality provides information about the state of a cluster and any user-defined workloads that you are monitoring. As a cluster administrator or as a user with view permissions for all projects, you can access metrics for all default OpenShift Container Platform and user-defined projects in the Metrics UI. The Metrics UI includes predefined queries, for example, CPU, memory, bandwidth, or network packet for all projects. You can also run custom Prometheus Query Language (PromQL) queries. Prerequisites You have access to the cluster as a user with the cluster-admin cluster role or with view permissions for all projects. You have installed the OpenShift CLI ( oc ). Procedure In the Administrator perspective of the OpenShift Container Platform web console, click Observe and go to the Metrics tab. To add one or more queries, perform any of the following actions: Option Description Select an existing query. From the Select query drop-down list, select an existing query. Create a custom query. Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item. Add multiple queries. Click Add query . Duplicate an existing query. Click the options menu to the query, then choose Duplicate query . Disable a query from being run. Click the options menu to the query and choose Disable query . To run queries that you created, click Run queries . The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message. Note When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs. By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query. Optional: Save the page URL to use this set of queries again in the future. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions: Option Description Hide all metrics from a query. Click the options menu for the query and click Hide all series . Hide a specific metric. Go to the query table and click the colored square near the metric name. Zoom into the plot and change the time range. Perform one of the following actions: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu to select the time range. Reset the time range. Click Reset zoom . Display outputs for all queries at a specific point in time. Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box. Hide the plot. Click Hide graph . Additional resources Querying Prometheus (Prometheus documentation) 5.1.3. Getting detailed information about a metrics target You can use the OpenShift Container Platform web console to view, search, and filter the endpoints that are currently targeted for scraping, which helps you to identify and troubleshoot problems. For example, you can view the current status of targeted endpoints to see when OpenShift Container Platform monitoring is not able to scrape metrics from a targeted component. The Metrics targets page shows targets for default OpenShift Container Platform projects and for user-defined projects. Prerequisites You have access to the cluster as an administrator for the project for which you want to view metrics targets. Procedure In the Administrator perspective of the OpenShift Container Platform web console, go to Observe Targets . The Metrics targets page opens with a list of all service endpoint targets that are being scraped for metrics. This page shows details about targets for default OpenShift Container Platform and user-defined projects. This page lists the following information for each target: Service endpoint URL being scraped The ServiceMonitor resource being monitored The up or down status of the target Namespace Last scrape time Duration of the last scrape Optional: To find a specific target, perform any of the following actions: Option Description Filter the targets by status and source. Choose filters in the Filter list. The following filtering options are available: Status filters: Up . The target is currently up and being actively scraped for metrics. Down . The target is currently down and not being scraped for metrics. Source filters: Platform . Platform-level targets relate only to default Red Hat OpenShift Service on AWS projects. These projects provide core Red Hat OpenShift Service on AWS functionality. User . User targets relate to user-defined projects. These projects are user-created and can be customized. Search for a target by name or label. Enter a search term in the Text or Label field to the search box. Sort the targets. Click one or more of the Endpoint Status , Namespace , Last Scrape , and Scrape Duration column headers. Click the URL in the Endpoint column for a target to go to its Target details page. This page provides information about the target, including the following information: The endpoint URL being scraped for metrics The current Up or Down status of the target A link to the namespace A link to the ServiceMonitor resource details Labels attached to the target The most recent time that the target was scraped for metrics 5.1.4. Reviewing monitoring dashboards as a cluster administrator In the Administrator perspective, you can view dashboards relating to core OpenShift Container Platform cluster components. Prerequisites You have access to the cluster as a user with the cluster-admin cluster role. Procedure In the Administrator perspective of the OpenShift Container Platform web console, go to Observe Dashboards . Choose a dashboard in the Dashboard list. Some dashboards, such as etcd and Prometheus dashboards, produce additional sub-menus when selected. Optional: Select a time range for the graphs in the Time Range list. Select a pre-defined time period. Set a custom time range by clicking Custom time range in the Time Range list. Input or select the From and To dates and times. Click Save to save the custom time range. Optional: Select a Refresh Interval . Hover over each of the graphs within a dashboard to display detailed information about specific items. Additional resources About monitoring dashboards 5.2. Accessing metrics as a developer You can access metrics to monitor the performance of your cluster workloads. Additional resources Understanding metrics 5.2.1. Viewing a list of available metrics As a cluster administrator or as a user with view permissions for all projects, you can view a list of metrics available in a cluster and output the list in JSON format. Prerequisites You are a cluster administrator, or you have access to the cluster as a user with the cluster-monitoring-view cluster role. You have installed the OpenShift Container Platform CLI ( oc ). You have obtained the OpenShift Container Platform API route for Thanos Querier. You are able to get a bearer token by using the oc whoami -t command. Important You can only use bearer token authentication to access the Thanos Querier API route. Procedure If you have not obtained the OpenShift Container Platform API route for Thanos Querier, run the following command: USD oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}' Retrieve a list of metrics in JSON format from the Thanos Querier API route by running the following command. This command uses oc to authenticate with a bearer token. USD curl -k -H "Authorization: Bearer USD(oc whoami -t)" https://<thanos_querier_route>/api/v1/metadata 1 1 Replace <thanos_querier_route> with the OpenShift Container Platform API route for Thanos Querier. 5.2.2. Querying metrics for user-defined projects with the OpenShift Container Platform web console You can use the OpenShift Container Platform metrics query browser to run Prometheus Query Language (PromQL) queries to examine metrics visualized on a plot. This functionality provides information about any user-defined workloads that you are monitoring. As a developer, you must specify a project name when querying metrics. You must have the required privileges to view metrics for the selected project. The Metrics UI includes predefined queries, for example, CPU, memory, bandwidth, or network packet. These queries are restricted to the selected project. You can also run custom Prometheus Query Language (PromQL) queries for the project. Note Developers can only use the Developer perspective and not the Administrator perspective. As a developer, you can only query metrics for one project at a time. Prerequisites You have access to the cluster as a developer or as a user with view permissions for the project that you are viewing metrics for. You have enabled monitoring for user-defined projects. You have deployed a service in a user-defined project. You have created a ServiceMonitor custom resource definition (CRD) for the service to define how the service is monitored. Procedure In the Developer perspective of the OpenShift Container Platform web console, click Observe and go to the Metrics tab. Select the project that you want to view metrics for from the Project: list. To add one or more queries, perform any of the following actions: Option Description Select an existing query. From the Select query drop-down list, select an existing query. Create a custom query. Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item. Add multiple queries. Click Add query . Duplicate an existing query. Click the options menu to the query, then choose Duplicate query . Disable a query from being run. Click the options menu to the query and choose Disable query . To run queries that you created, click Run queries . The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message. Note When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs. By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query. Optional: Save the page URL to use this set of queries again in the future. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions: Option Description Hide all metrics from a query. Click the options menu for the query and click Hide all series . Hide a specific metric. Go to the query table and click the colored square near the metric name. Zoom into the plot and change the time range. Perform one of the following actions: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu to select the time range. Reset the time range. Click Reset zoom . Display outputs for all queries at a specific point in time. Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box. Hide the plot. Click Hide graph . Additional resources Querying Prometheus (Prometheus documentation) 5.2.3. Reviewing monitoring dashboards as a developer In the Developer perspective, you can view dashboards relating to a selected project. Note In the Developer perspective, you can view dashboards for only one project at a time. Prerequisites You have access to the cluster as a developer or as a user. You have view permissions for the project that you are viewing the dashboard for. Procedure In the Developer perspective in the OpenShift Container Platform web console, click Observe and go to the Dashboards tab. Select a project from the Project: drop-down list. Select a dashboard from the Dashboard drop-down list to see the filtered metrics. Note All dashboards produce additional sub-menus when selected, except Kubernetes / Compute Resources / Namespace (Pods) . Optional: Select a time range for the graphs in the Time Range list. Select a pre-defined time period. Set a custom time range by clicking Custom time range in the Time Range list. Input or select the From and To dates and times. Click Save to save the custom time range. Optional: Select a Refresh Interval . Hover over each of the graphs within a dashboard to display detailed information about specific items. Additional resources About monitoring dashboards Monitoring project and application metrics using the Developer perspective 5.3. Accessing monitoring APIs by using the CLI In OpenShift Container Platform, you can access web service APIs for some monitoring components from the command-line interface (CLI). Important In certain situations, accessing API endpoints can degrade the performance and scalability of your cluster, especially if you use endpoints to retrieve, send, or query large amounts of metrics data. To avoid these issues, consider the following recommendations: Avoid querying endpoints frequently. Limit queries to a maximum of one every 30 seconds. Do not retrieve all metrics data through the /federate endpoint for Prometheus. Query the endpoint only when you want to retrieve a limited, aggregated data set. For example, retrieving fewer than 1,000 samples for each request helps minimize the risk of performance degradation. 5.3.1. About accessing monitoring web service APIs You can directly access web service API endpoints from the command line for the following monitoring stack components: Prometheus Alertmanager Thanos Ruler Thanos Querier Important To access Thanos Ruler and Thanos Querier service APIs, the requesting account must have get permission on the namespaces resource, which can be granted by binding the cluster-monitoring-view cluster role to the account. When you access web service API endpoints for monitoring components, be aware of the following limitations: You can only use bearer token authentication to access API endpoints. You can only access endpoints in the /api path for a route. If you try to access an API endpoint in a web browser, an Application is not available error occurs. To access monitoring features in a web browser, use the OpenShift Container Platform web console to review monitoring dashboards. Additional resources Reviewing monitoring dashboards as a cluster administrator Reviewing monitoring dashboards as a developer 5.3.2. Accessing a monitoring web service API The following example shows how to query the service API receivers for the Alertmanager service used in core platform monitoring. You can use a similar method to access the prometheus-k8s service for core platform Prometheus and the thanos-ruler service for Thanos Ruler. Prerequisites You are logged in to an account that is bound against the monitoring-alertmanager-edit role in the openshift-monitoring namespace. You are logged in to an account that has permission to get the Alertmanager API route. Note If your account does not have permission to get the Alertmanager API route, a cluster administrator can provide the URL for the route. Procedure Extract an authentication token by running the following command: USD TOKEN=USD(oc whoami -t) Extract the alertmanager-main API route URL by running the following command: USD HOST=USD(oc -n openshift-monitoring get route alertmanager-main -ojsonpath='{.status.ingress[].host}') Query the service API receivers for Alertmanager by running the following command: USD curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v2/receivers" 5.3.3. Querying metrics by using the federation endpoint for Prometheus You can use the federation endpoint for Prometheus to scrape platform and user-defined metrics from a network location outside the cluster. To do so, access the Prometheus /federate endpoint for the cluster via an OpenShift Container Platform route. Important A delay in retrieving metrics data occurs when you use federation. This delay can affect the accuracy and timeliness of the scraped metrics. Using the federation endpoint can also degrade the performance and scalability of your cluster, especially if you use the federation endpoint to retrieve large amounts of metrics data. To avoid these issues, follow these recommendations: Do not try to retrieve all metrics data via the federation endpoint for Prometheus. Query it only when you want to retrieve a limited, aggregated data set. For example, retrieving fewer than 1,000 samples for each request helps minimize the risk of performance degradation. Avoid frequent querying of the federation endpoint for Prometheus. Limit queries to a maximum of one every 30 seconds. If you need to forward large amounts of data outside the cluster, use remote write instead. For more information, see the Configuring remote write storage section. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-monitoring-view cluster role or have obtained a bearer token with get permission on the namespaces resource. Note You can only use bearer token authentication to access the Prometheus federation endpoint. You are logged in to an account that has permission to get the Prometheus federation route. Note If your account does not have permission to get the Prometheus federation route, a cluster administrator can provide the URL for the route. Procedure Retrieve the bearer token by running the following the command: USD TOKEN=USD(oc whoami -t) Get the Prometheus federation route URL by running the following command: USD HOST=USD(oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath='{.status.ingress[].host}') Query metrics from the /federate route. The following example command queries up metrics: USD curl -G -k -H "Authorization: Bearer USDTOKEN" https://USDHOST/federate --data-urlencode 'match[]=up' Example output # TYPE up untyped up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834 ... 5.3.4. Accessing metrics from outside the cluster for custom applications You can query Prometheus metrics from outside the cluster when monitoring your own services with user-defined projects. Access this data from outside the cluster by using the thanos-querier route. This access only supports using a bearer token for authentication. Prerequisites You have deployed your own service, following the "Enabling monitoring for user-defined projects" procedure. You are logged in to an account with the cluster-monitoring-view cluster role, which provides permission to access the Thanos Querier API. You are logged in to an account that has permission to get the Thanos Querier API route. Note If your account does not have permission to get the Thanos Querier API route, a cluster administrator can provide the URL for the route. Procedure Extract an authentication token to connect to Prometheus by running the following command: USD TOKEN=USD(oc whoami -t) Extract the thanos-querier API route URL by running the following command: USD HOST=USD(oc -n openshift-monitoring get route thanos-querier -ojsonpath='{.status.ingress[].host}') Set the namespace to the namespace in which your service is running by using the following command: USD NAMESPACE=ns1 Query the metrics of your own services in the command line by running the following command: USD curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v1/query?" --data-urlencode "query=up{namespace='USDNAMESPACE'}" The output shows the status for each application pod that Prometheus is scraping: The formatted example output { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "up", "endpoint": "web", "instance": "10.129.0.46:8080", "job": "prometheus-example-app", "namespace": "ns1", "pod": "prometheus-example-app-68d47c4fb6-jztp2", "service": "prometheus-example-app" }, "value": [ 1591881154.748, "1" ] } ], } } Note The formatted example output uses a filtering tool, such as jq , to provide the formatted indented JSON. See the jq Manual (jq documentation) for more information about using jq . The command requests an instant query endpoint of the Thanos Querier service, which evaluates selectors at one point in time. 5.3.5. Resources reference for the Cluster Monitoring Operator This document describes the following resources deployed and managed by the Cluster Monitoring Operator (CMO): Routes Services Use this information when you want to configure API endpoint connections to retrieve, send, or query metrics data. Important In certain situations, accessing endpoints can degrade the performance and scalability of your cluster, especially if you use endpoints to retrieve, send, or query large amounts of metrics data. To avoid these issues, follow these recommendations: Avoid querying endpoints frequently. Limit queries to a maximum of one every 30 seconds. Do not try to retrieve all metrics data via the /federate endpoint. Query it only when you want to retrieve a limited, aggregated data set. For example, retrieving fewer than 1,000 samples for each request helps minimize the risk of performance degradation. 5.3.5.1. CMO routes resources 5.3.5.1.1. openshift-monitoring/alertmanager-main Expose the /api endpoints of the alertmanager-main service via a router. 5.3.5.1.2. openshift-monitoring/prometheus-k8s Expose the /api endpoints of the prometheus-k8s service via a router. 5.3.5.1.3. openshift-monitoring/prometheus-k8s-federate Expose the /federate endpoint of the prometheus-k8s service via a router. 5.3.5.1.4. openshift-user-workload-monitoring/federate Expose the /federate endpoint of the prometheus-user-workload service via a router. 5.3.5.1.5. openshift-monitoring/thanos-querier Expose the /api endpoints of the thanos-querier service via a router. 5.3.5.1.6. openshift-user-workload-monitoring/thanos-ruler Expose the /api endpoints of the thanos-ruler service via a router. 5.3.5.2. CMO services resources 5.3.5.2.1. openshift-monitoring/prometheus-operator-admission-webhook Expose the admission webhook service which validates PrometheusRules and AlertmanagerConfig custom resources on port 8443. 5.3.5.2.2. openshift-user-workload-monitoring/alertmanager-user-workload Expose the user-defined Alertmanager web server within the cluster on the following ports: Port 9095 provides access to the Alertmanager endpoints. Granting access requires binding a user to the monitoring-alertmanager-api-reader role (for read-only operations) or monitoring-alertmanager-api-writer role in the openshift-user-workload-monitoring project. Port 9092 provides access to the Alertmanager endpoints restricted to a given project. Granting access requires binding a user to the monitoring-rules-edit cluster role or monitoring-edit cluster role in the project. Port 9097 provides access to the /metrics endpoint only. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.3. openshift-monitoring/alertmanager-main Expose the Alertmanager web server within the cluster on the following ports: Port 9094 provides access to all the Alertmanager endpoints. Granting access requires binding a user to the monitoring-alertmanager-view (for read-only operations) or monitoring-alertmanager-edit role in the openshift-monitoring project. Port 9092 provides access to the Alertmanager endpoints restricted to a given project. Granting access requires binding a user to the monitoring-rules-edit cluster role or monitoring-edit cluster role in the project. Port 9097 provides access to the /metrics endpoint only. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.4. openshift-monitoring/kube-state-metrics Expose kube-state-metrics /metrics endpoints within the cluster on the following ports: Port 8443 provides access to the Kubernetes resource metrics. This port is for internal use, and no other usage is guaranteed. Port 9443 provides access to the internal kube-state-metrics metrics. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.5. openshift-monitoring/metrics-server Expose the metrics-server web server on port 443. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.6. openshift-monitoring/monitoring-plugin Expose the monitoring plugin service on port 9443. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.7. openshift-monitoring/node-exporter Expose the /metrics endpoint on port 9100. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.8. openshift-monitoring/openshift-state-metrics Expose openshift-state-metrics /metrics endpoints within the cluster on the following ports: Port 8443 provides access to the OpenShift resource metrics. This port is for internal use, and no other usage is guaranteed. Port 9443 provides access to the internal openshift-state-metrics metrics. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.9. openshift-monitoring/prometheus-k8s Expose the Prometheus web server within the cluster on the following ports: Port 9091 provides access to all the Prometheus endpoints. Granting access requires binding a user to the cluster-monitoring-view cluster role. Port 9092 provides access to the /metrics and /federate endpoints only. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.10. openshift-user-workload-monitoring/prometheus-operator Expose the /metrics endpoint on port 8443. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.11. openshift-monitoring/prometheus-operator Expose the /metrics endpoint on port 8443. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.12. openshift-user-workload-monitoring/prometheus-user-workload Expose the Prometheus web server within the cluster on the following ports: Port 9091 provides access to the /metrics endpoint only. This port is for internal use, and no other usage is guaranteed. Port 9092 provides access to the /federate endpoint only. Granting access requires binding a user to the cluster-monitoring-view cluster role. This also exposes the /metrics endpoint of the Thanos sidecar web server on port 10902. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.13. openshift-monitoring/telemeter-client Expose the /metrics endpoint on port 8443. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.14. openshift-monitoring/thanos-querier Expose the Thanos Querier web server within the cluster on the following ports: Port 9091 provides access to all the Thanos Querier endpoints. Granting access requires binding a user to the cluster-monitoring-view cluster role. Port 9092 provides access to the /api/v1/query , /api/v1/query_range/ , /api/v1/labels , /api/v1/label/*/values , and /api/v1/series endpoints restricted to a given project. Granting access requires binding a user to the view cluster role in the project. Port 9093 provides access to the /api/v1/alerts , and /api/v1/rules endpoints restricted to a given project. Granting access requires binding a user to the monitoring-rules-edit , monitoring-edit , or monitoring-rules-view cluster role in the project. Port 9094 provides access to the /metrics endpoint only. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.15. openshift-user-workload-monitoring/thanos-ruler Expose the Thanos Ruler web server within the cluster on the following ports: Port 9091 provides access to all Thanos Ruler endpoints. Granting access requires binding a user to the cluster-monitoring-view cluster role. Port 9092 provides access to the /metrics endpoint only. This port is for internal use, and no other usage is guaranteed. This also exposes the gRPC endpoints on port 10901. This port is for internal use, and no other usage is guaranteed. 5.3.5.2.16. openshift-monitoring/cluster-monitoring-operator Expose the /metrics and /validate-webhook endpoints on port 8443. This port is for internal use, and no other usage is guaranteed. 5.3.6. Additional resources Enabling monitoring for user-defined projects Configuring remote write storage for core platform monitoring Configuring remote write storage for monitoring of user-defined projects Accessing metrics as an administrator Accessing metrics as a developer Managing alerts as an Administrator Managing alerts as a Developer

Commands:
oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}'
curl -k -H "Authorization: Bearer USD(oc whoami -t)" https://<thanos_querier_route>/api/v1/metadata 1
oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}'
curl -k -H "Authorization: Bearer USD(oc whoami -t)" https://<thanos_querier_route>/api/v1/metadata 1
TOKEN=USD(oc whoami -t)
HOST=USD(oc -n openshift-monitoring get route alertmanager-main -ojsonpath='{.status.ingress[].host}')
curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v2/receivers"
TOKEN=USD(oc whoami -t)
HOST=USD(oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath='{.status.ingress[].host}')
curl -G -k -H "Authorization: Bearer USDTOKEN" https://USDHOST/federate --data-urlencode 'match[]=up'
TYPE up untyped up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834
TOKEN=USD(oc whoami -t)
HOST=USD(oc -n openshift-monitoring get route thanos-querier -ojsonpath='{.status.ingress[].host}')
NAMESPACE=ns1
curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v1/query?" --data-urlencode "query=up{namespace='USDNAMESPACE'}"
{ "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "up", "endpoint": "web", "instance": "10.129.0.46:8080", "job": "prometheus-example-app", "namespace": "ns1", "pod": "prometheus-example-app-68d47c4fb6-jztp2", "service": "prometheus-example-app" }, "value": [ 1591881154.748, "1" ] } ], } }