Project: CloudNativePG (App Definition and Development / Database)
File: monitoring.md

!!! Important Installing Prometheus and Grafana is beyond the scope of this project. We assume they are correctly installed in your system. However, for experimentation we provide instructions in . For each PostgreSQL instance, the operator provides an exporter of metrics for via HTTP, on port 9187, named `metrics`. The operator comes with a , as well as a highly configurable and customizable system to define additional queries via one or more `ConfigMap` or `Secret` resources (see the below for details). !!! Important Starting from version 1.11, CloudNativePG already installs in a `ConfigMap` called `default-monitoring`. !!! Info You can inspect the exported metrics by following the instructions in the section below. All monitoring queries that are performed on PostgreSQL are: atomic (one transaction per query) executed with the `pg_monitor` role executed with `applicationname` set to `cnpgmetrics_exporter` executed as user `postgres` Please refer to the "Predefined Roles" section in PostgreSQL for details on the `pg_monitor` role. Queries, by default, are run against the main database, as defined by the specified `bootstrap` method of the `Cluster` resource, according to the following logic: using `initdb`: queries will be run by default against the specified database in `initdb.database`, or `app` if not specified using `recovery`: queries will be run by default against the specified database in `recovery.database`, or `postgres` if not specified using `pg_basebackup`: queries will be run by default against the specified database in `pg_basebackup.database`, or `postgres` if not specified The default database can always be overridden for a given user-defined metric, by specifying a list of one or more databases in the `target_databases` option. !!! Seealso "Prometheus/Grafana" If you are interested in evaluating the integration of CloudNativePG with Prometheus and Grafana, you can find a quick setup guide in A specific PostgreSQL cluster can be monitored using the resource . A PodMonitor correctly pointing to a Cluster can be automatically created by the operator by setting `.spec.monitoring.enablePodMonitor` to `true` in the Cluster resource itself (default: false). !!! Important Any change to the `PodMonitor` created automatically will be overridden by the Operator at the next reconciliation cycle, in case you need to customize it, you can do so as described below. To deploy a `PodMonitor` for a specific Cluster manually, you can just define it as follows, changing it as needed: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: cluster-example spec: selector: matchLabels: "cnpg.io/cluster": cluster-example podMetricsEndpoints: port: metrics ``` !!! Important Make sure you modify the example above with a unique name as well as the correct cluster's namespace and labels (we are using `cluster-example`). !!! Important Label `postgresql`, used in previous versions of this document, is deprecated and will be removed in the future. Please use the label `cnpg.io/cluster` instead to select the

Every PostgreSQL instance exporter automatically exposes a set of predefined metrics, which can be classified in two major categories: PostgreSQL related metrics, starting with `cnpgcollector*`, including: number of WAL files and total size on disk number of `.ready` and `.done` files in the archive status folder requested minimum and maximum number of synchronous replicas, as well as the expected and actually observed values number of distinct nodes accommodating the instances timestamps indicating last failed and last available backup, as well as the first point of recoverability for the cluster flag indicating if replica cluster mode is enabled or disabled flag indicating if a manual switchover is required flag indicating if fencing is enabled or disabled Go runtime related metrics, starting with `go_*` Below is a sample of the metrics returned by the `localhost:9187/metrics` endpoint of an instance. As you can see, the Prometheus format is self-documenting: ```text cnpgcollectorcollectiondurationseconds{collector="Collect.up"} 0.0031393 cnpgcollectorcollections_total 2 cnpgcollectorfencing_on 0 cnpgcollectornodes_used 3 cnpgcollectorlastcollectionerror 0 cnpgcollectormanualswitchoverrequired 0 cnpgcollectorpg_wal{value="count"} 9 cnpgcollectorpgwal{value="slotsmax"} NaN cnpgcollectorpg_wal{value="keep"} 32 cnpgcollectorpg_wal{value="max"} 64 cnpgcollectorpg_wal{value="min"} 5 cnpgcollectorpg_wal{value="size"} 1.50994944e+08 cnpgcollectorpgwal{value="volumemax"} 128 cnpgcollectorpgwal{value="volumesize"} 2.147483648e+09 cnpgcollectorpgwalarchive_status{value="done"} 6 cnpgcollectorpgwalarchive_status{value="ready"} 0 cnpgcollectorreplica_mode 0 cnpgcollectorsync_replicas{value="expected"} 0 cnpgcollectorsync_replicas{value="max"} 0 cnpgcollectorsync_replicas{value="min"} 0 cnpgcollectorsync_replicas{value="observed"} 0 cnpgcollectorup{cluster="cluster-example"} 1 cnpgcollectorpostgres_version{cluster="cluster-example",full="16.3"} 16.3 cnpgcollectorlastfailedbackup_timestamp 0 cnpgcollectorlastavailablebackup_timestamp 1.63238406e+09 cnpgcollectorfirstrecoverabilitypoint 1.63238406e+09 cnpgcollectorlo_pages{datname="app"} 0 cnpgcollectorlo_pages{datname="postgres"} 78 cnpgcollectorwalbuffersfull{stats_reset="2023-06-19T10:51:27.473259Z"} 6472 cnpgcollectorwalbytes{statsreset="2023-06-19T10:51:27.473259Z"} 1.0035147e+07 cnpgcollectorwalfpi{statsreset="2023-06-19T10:51:27.473259Z"} 1474 cnpgcollectorwalrecords{statsreset="2023-06-19T10:51:27.473259Z"} 26178 cnpgcollectorwalsync{statsreset="2023-06-19T10:51:27.473259Z"} 37 cnpgcollectorwalsynctime{stats_reset="2023-06-19T10:51:27.473259Z"} 0 cnpgcollectorwalwrite{statsreset="2023-06-19T10:51:27.473259Z"} 7243 cnpgcollectorwalwritetime{stats_reset="2023-06-19T10:51:27.473259Z"} 0 cnpglasterror 0 gogcduration_seconds{quantile="0"} 5.01e-05 gogcduration_seconds{quantile="0.25"} 7.27e-05 gogcduration_seconds{quantile="0.5"} 0.0001748 gogcduration_seconds{quantile="0.75"} 0.0002959 gogcduration_seconds{quantile="1"} 0.0012776 gogcdurationsecondssum 0.0035741 gogcdurationsecondscount 13 go_goroutines 25 go_info{version="go1.20.5"} 1 gomemstatsalloc_bytes 4.493744e+06 gomemstatsallocbytestotal 2.1698216e+07 gomemstatsbuckhashsys_bytes 1.456234e+06 gomemstatsfrees_total 172118 gomemstatsgccpufraction 1.0749468700447189e-05 gomemstatsgcsysbytes 5.530048e+06 gomemstatsheapallocbytes 4.493744e+06 gomemstatsheapidlebytes 5.8236928e+07 gomemstatsheapinusebytes 7.528448e+06 gomemstatsheap_objects 26306 gomemstatsheapreleasedbytes 5.7401344e+07 gomemstatsheapsysbytes 6.5765376e+07 gomemstatslastgctime_seconds 1.6311727586032727e+09 gomemstatslookups_total 0 gomemstatsmallocs_total 198424 gomemstatsmcacheinusebytes 14400 gomemstatsmcachesysbytes 16384 gomemstatsmspaninusebytes 191896 gomemstatsmspansysbytes 212992 gomemstatsnextgcbytes 8.689632e+06 gomemstatsothersysbytes 2.566622e+06 gomemstatsstackinusebytes 1.343488e+06 gomemstatsstacksysbytes 1.343488e+06 gomemstatssys_bytes 7.6891144e+07 go_threads 18 ``` !!! Note `cnpgcollectorpostgres_version` is a GaugeVec metric containing the `Major.Minor` version of PostgreSQL. The full semantic version `Major.Minor.Patch` can be found inside one of its label field named `full`. !!! Note `cnpgcollectorfirstrecoverabilitypoint` and `cnpgcollectorlastavailablebackup_timestamp` will be zero until your first backup to the object store. This is separate from the WAL archival. This feature is currently in beta state and the format is inspired by the of the PostgreSQL Prometheus Exporter. Custom metrics can be defined by users by referring to the created `Configmap`/`Secret` in a `Cluster` definition under the `.spec.monitoring.customQueriesConfigMap` or `customQueriesSecret` section as in the following example: ```yaml apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: cluster-example namespace: test spec: instances: 3 storage: size: 1Gi monitoring: customQueriesConfigMap: name: example-monitoring key: custom-queries ``` The `customQueriesConfigMap`/`customQueriesSecret` sections contain a list of `ConfigMap`/`Secret` references specifying the key in which the custom queries are defined. Take care that the referred resources have to be created in the same namespace as the Cluster resource. !!! Note If you want ConfigMaps and Secrets to be automatically reloaded by instances, you can add a label with key `cnpg.io/reload` to it, otherwise you will have to reload the instances using the `kubectl cnpg reload`

!!! Important When a user defined metric overwrites an already existing metric the instance manager prints a json warning log, containing the message:`Query with the same name already found. Overwriting the existing one.` and a key `queryName` containing the overwritten query name. Here you can see an example of a `ConfigMap` containing a single custom query, referenced by the `Cluster` example above: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: example-monitoring namespace: test labels: cnpg.io/reload: "" data: custom-queries: | pg_replication: query: "SELECT CASE WHEN NOT pgisin_recovery() THEN 0 ELSE GREATEST (0, EXTRACT(EPOCH FROM (now() - pglastxactreplaytimestamp()))) END AS lag, pgisinrecovery() AS inrecovery, EXISTS (TABLE pgstatwalreceiver) AS iswalreceiverup, (SELECT count(*) FROM pgstatreplication) AS streaming_replicas" metrics: lag: usage: "GAUGE" description: "Replication lag behind primary in seconds" in_recovery: usage: "GAUGE" description: "Whether the instance is in recovery" iswalreceiver_up: usage: "GAUGE" description: "Whether the instance wal_receiver is up" streaming_replicas: usage: "GAUGE" description: "Number of streaming replicas connected to the instance" ``` A list of basic monitoring queries can be found in the that is already installed in your CloudNativePG deployment (see ). If the `target_databases` option lists more than one database the metric is collected from each of them. Database auto-discovery can be enabled for a specific query by specifying a shell-like pattern (i.e., containing `*`, `?` or `[]`) in the list of `target_databases`. If provided, the operator will expand the list of target databases by adding all the databases returned by the execution of `SELECT datname FROM pg_database WHERE datallowconn AND NOT datistemplate` and matching the pattern according to rules. !!! Note The `*` character has a in yaml, so you need to quote (`"*"`) the `target_databases` value when it includes such a

It is recommended that you always include the name of the database in the returned labels, for example using the `current_database()` function as in the following example: ```yaml some_query: | query: | SELECT current_database() as datname, count(*) as rows FROM some_table metrics: datname: usage: "LABEL" description: "Name of current database" rows: usage: "GAUGE" description: "number of rows" target_databases: albert bb freddie ``` This will produce in the following metric being exposed: ```text cnpgsomequery_rows{datname="albert"} 2 cnpgsomequery_rows{datname="bb"} 5 cnpgsomequery_rows{datname="freddie"} 10 ``` Here is an example of a query with auto-discovery enabled which also runs on the `template1` database (otherwise not returned by the aforementioned query): ```yaml some_query: | query: | SELECT current_database() as datname, count(*) as rows FROM some_table metrics: datname: usage: "LABEL" description: "Name of current database" rows: usage: "GAUGE" description: "number of rows" target_databases: "*" "template1" ``` The above example will produce the following metrics (provided the databases exist): ```text cnpgsomequery_rows{datname="albert"} 2 cnpgsomequery_rows{datname="bb"} 5 cnpgsomequery_rows{datname="freddie"} 10 cnpgsomequery_rows{datname="template1"} 7 cnpgsomequery_rows{datname="postgres"} 42 ``` Every custom query has the following basic structure: ```yaml <MetricName>: query: "<SQLQuery>" metrics: <ColumnName>: usage: "<MetricType>" description: "<MetricDescription>" ``` Here is a short description of all the available fields: `<MetricName>`: the name of the Prometheus metric `query`: the SQL query to run on the target database to generate the metrics `primary`: whether to run the query only on the primary instance `master`: same as `primary` (for compatibility with the Prometheus PostgreSQL exporter's syntax - deprecated) <!-- wokeignore:rule=master --> `runonserver`: a semantic version range to limit the versions of PostgreSQL the query should run on (e.g. `">=11.0.0"` or `">=12.0.0 <=15.0.0"`) `target_databases`: a list of databases to run the `query` against, or a to enable auto discovery. Overwrites the default database if provided. `metrics`: section containing a list of all exported columns, defined as follows: `<ColumnName>`: the name of the column returned by the query `usage`: one of the values described below `description`: the metric's description `metrics_mapping`: the optional column mapping when `usage` is set to `MAPPEDMETRIC` The possible values for `usage` are: | Column Usage Label | Description | |:--|:| | `DISCARD` | this column should be ignored | | `LABEL` | use this column as a label | | `COUNTER` | use this column as a counter | | `GAUGE` | use this column as a gauge | | `MAPPEDMETRIC` | use this column with the supplied mapping of text values | | `DURATION` | use this column as a text duration (in milliseconds) | | `HISTOGRAM` | use this column as a histogram | Please visit the from the Prometheus documentation for more information. Custom defined metrics are returned by the Prometheus exporter endpoint (`:9187/metrics`) with the following format: ```text cnpg<MetricName><ColumnName>{<LabelColumnName>=<LabelColumnValue> ... } <ColumnValue> ``` !!! Note `LabelColumnName` are metrics with `usage` set to `LABEL` and their `Value` Considering the `pg_replication` example above, the exporter's endpoint would return the following output when invoked: ```text cnpgpgreplicationinrecovery 0 cnpgpgreplication_lag 0 cnpgpgreplicationstreamingreplicas 2 cnpgpgreplicationiswalreceiverup 0 ``` The operator can be configured to automatically inject in a Cluster a set of monitoring queries defined in a ConfigMap or a Secret, inside the operator's namespace. You have to set the `MONITORINGQUERIESCONFIGMAP` or `MONITORINGQUERIESSECRET` key in the , respectively to the name of the ConfigMap or the Secret; the operator will then use the content of the `queries` key. Any change to the `queries` content will be immediately reflected on all the deployed Clusters using it. The operator installation manifests come with a predefined ConfigMap, called `cnpg-default-monitoring`, to be used by all Clusters. `MONITORINGQUERIESCONFIGMAP` is by default set to `cnpg-default-monitoring` in the operator configuration. If you want to disable the default set of metrics, you can: disable it at operator level: set the `MONITORINGQUERIESCONFIGMAP`/`MONITORINGQUERIESSECRET` key to `""` (empty string), in the operator ConfigMap. Changes to operator ConfigMap require an operator restart. disable it for a specific Cluster: set `.spec.monitoring.disableDefaultQueries` to `true` in the Cluster. !!! Important The ConfigMap or Secret specified via `MONITORINGQUERIESCONFIGMAP`/`MONITORINGQUERIESSECRET` will always be copied to the Cluster's namespace with a fixed name: `cnpg-default-monitoring`. So that, if you intend to have default metrics, you should not create a ConfigMap with this name in the cluster's namespace. CloudNativePG is inspired by the PostgreSQL Prometheus Exporter, but presents some differences. In particular, the `cache_seconds` field is not implemented in CloudNativePG's

The operator internally exposes metrics via HTTP on port 8080, named `metrics`. !!! Info You can inspect the exported metrics by following the instructions in the section below. Currently, the operator exposes default `kubebuilder` metrics, see for more details. The operator deployment can be monitored using the by defining the following resource: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: cnpg-controller-manager spec: selector: matchLabels: app.kubernetes.io/name: cloudnative-pg podMetricsEndpoints: port: metrics ``` In this section we provide some basic instructions on how to inspect the metrics exported by a specific PostgreSQL instance manager (primary or replica) or the operator, using a temporary pod running `curl` in the same namespace. !!! Note In the example below we assume we are working in the default namespace, alongside with the PostgreSQL cluster. Please feel free to adapt this example to your use case, by applying basic Kubernetes knowledge. Create the `curl.yaml` file with this content: ```yaml apiVersion: v1 kind: Pod metadata: name: curl spec: containers: name: curl image: curlimages/curl:8.2.1 command: ['sleep', '3600'] ``` Then create the pod: ```shell kubectl apply -f curl.yaml ``` In case you want to inspect the metrics exported by an instance, you need to connect to port 9187 of the target pod. This is the generic command to be run (make sure you use the correct IP for the pod): ```shell kubectl exec -ti curl -- curl -s <pod_ip>:9187/metrics ``` For example, if your PostgreSQL cluster is called `cluster-example` and you want to retrieve the exported metrics of the first pod in the cluster, you can run the following command to programmatically get the IP of that pod: ```shell POD_IP=$(kubectl get pod cluster-example-1 --template '{{.status.podIP}}') ``` And then run: ```shell kubectl exec -ti curl -- curl -s ${POD_IP}:9187/metrics ``` In case you want to access the metrics of the operator, you need to point to the pod where the operator is running, and use TCP port 8080 as target. At the end of the inspection, please make sure you delete the `curl` pod: ```shell kubectl delete -f curl.yaml ``` !!! Important These resources are provided for illustration and experimentation, and do not represent any kind of recommendation for your production system In the directory you will find a series of sample files for observability. Please refer to section for context: `kube-stack-config.yaml`: a configuration file for the kube-stack helm chart installation. It ensures that Prometheus listens for all PodMonitor resources. `prometheusrule.yaml`: a `PrometheusRule` with alerts for CloudNativePG. NOTE: this does not include inter-operation with notification services. Please refer to the . `podmonitor.yaml`: a `PodMonitor` for the CloudNativePG Operator deployment. In addition, we provide the "raw" sources for the Prometheus alert rules in the `alerts.yaml` file. The has a dedicated repository now. Note that, for the configuration of `kube-prometheus-stack`, other fields and settings are available over what we provide in `kube-stack-config.yaml`. You can execute `helm show values prometheus-community/kube-prometheus-stack` to view them. For further information, please refer to the page.