Project: YugabyteDB (App Definition and Development / Database)
File: alert-policy-templates.md

title: Alert policy templates headerTitle: Alert policy templates linkTitle: Alert policy templates description: Alert policy template reference menu: v2.18_yugabyte-platform: identifier: alert-policy-templates parent: set-up-alerts-health-checking weight: 40 type: docs use the following templates to define how the alert is triggered. The alert templates have been created using Prometheus expressions. CQLSH connection failure has been detected for universe `'$universe_name'` on `$value` T-Server instances. ```promql ybphealthcheckcqlshconnectivityerror{universeuuid="$uuid"} > 0 ``` Database compaction rejections detected for universe `'$universe_name'`. ```promql sum by (nodeprefix) (increase(majoritysstfilesrejections{nodeprefix="$nodeprefix"}[10m])) > 0 ``` Test YSQL write/read operation failed on `$value` nodes for universe `'$universe_name'`. ```promql count by (nodeprefix) (ybnodeysqlwriteread{nodeprefix="$node_prefix"} < 1) ``` Version mismatch has been detected for universe `'$universe_name'` for `$value` Master or T-Server instances. ```promql ybphealthchecktserverversionmismatch{universeuuid="$uuid"} + ybphealthcheckmasterversionmismatch{universeuuid="$uuid"} > 0 ``` Client to node certificate for universe `'$universe_name'` expires in `$value` days. ```promql min by (nodename) (ybphealthcheckc2ncertvaliditydays{universeuuid="$uuid"} < 30) ``` Failed to issue health check notification for universe `'$universe_name'`. You need to check Health notification settings and YugabyteDB Anywhere logs for details or contact {{% support-platform %}}. ```promql lastovertime(ybphealthchecknotificationstatus{universe_uuid = "$uuid"}[1d]) < 1 ``` Failed to perform health check for universe `'$universe_name'`. You need to check YugabyteDB Anywhere logs for details or contact {{% support-platform %}}. ```promql lastovertime(ybphealthcheckstatus{universeuuid = "$uuid"}[1d]) < 1 ``` Encryption at rest configuration for universe `'$universe_name'` expires in `$value` days. ```promql ybpuniverseencryptionkeyexpirydays{universeuuid="$uuid"} < 3 ``` Redis connection failure has been detected for universe `'$universe_name'` on `$value` T-Server instances. ```promql ybphealthcheckredisconnectivityerror{universeuuid="$uuid"} > 0 ``` Last attempt to run a scheduled backup for universe `'$universe_name'` failed due to other backup or universe operation in progress. ```promql lastovertime(ybpschedulebackupstatus{universeuuid = "$uuid"}[1d]) < 1 ``` Node to node certificate for universe `'$universe_name'` expires in `$value` days. ```promql min by (nodename) (ybphealthcheckn2ncertvaliditydays{universeuuid="$uuid"} < 30) ``` Core files detected for universe `'$universe_name'` on `$value` T-Server instances. ```promql ybphealthchecktservercorefiles{universeuuid="$uuid"} > 0 ``` Last attempt to send alert notifications for customer `'yugabyte support'` failed. You need to check YugabyteDB Anywhere logs for details or contact {{% support-platform %}}. ```promql lastovertime(ybpalertmanagerstatus{customeruuid = "$uuid"}[1d]) < 1 ``` Last backup task for universe `'$universe_name'` failed. You need to check the backup task result for details. ```promql lastovertime(ybpcreatebackupstatus{universeuuid = "$uuid"}[1d]) < 1 ``` `$value` tablets remain under-replicated for more than 5 minutes in universe `'$universe_name'`. ```promql max by (nodeprefix) (count by (nodeprefix, exportedinstance) (maxovertime(ybnodeunderreplicatedtablet{nodeprefix="$nodeprefix"}[5m])) > 0) ``` Database memory rejections have been detected for universe `'$universe_name'`. ```promql sum by (nodeprefix) (increase(leadermemorypressurerejections{nodeprefix="$nodeprefix"}[10m])) + sum by (nodeprefix) (increase(followermemorypressurerejections{nodeprefix="$nodeprefix"}[10m])) + sum by (nodeprefix) (increase(operationmemorypressurerejections{nodeprefix="$nodeprefix"}[10m])) > 0 ``` Client to node CA certificate for universe `'$universe_name'` expires in `$value` days. ```promql min by (nodename) (ybphealthcheckc2ncacertvaliditydays{universe_uuid="$uuid"} < 30) ``` Database queues overflow has been detected for universe `'$universe_name'`. ```promql sum by (nodeprefix) (increase(rpcsqueueoverflow{nodeprefix="$nodeprefix"}[10m])) + sum by (nodeprefix) (increase(rpcstimedoutinqueue{nodeprefix="$nodeprefix"}[10m])) > 1 ``` Last alert rules synchronization for customer `'yugabyte support'` has failed. YugabyteDB Anywhere logs for details or contact {{% support-platform

```promql lastovertime(ybpalertconfigwriterstatus[1d]) < 1 ``` Last alert query for customer `'yugabyte support'` failed. YugabyteDB Anywhere logs for details or contact {{% support-platform %}}. ```promql lastovertime(ybpalertquery_status[1d]) < 1 ``` Fatal logs have been detected for universe `'$universe_name'` on `$value` Master or T-Server instances. ```promql sum by (universeuuid) (ybphealthchecknodemasterfatallogs{universeuuid="$uuid"} < bool 1) + sum by (universeuuid) (ybphealthchecknodetserverfatallogs{universeuuid="$uuid"} < bool 1) > 0 ``` Master leader is missing for universe `'$universe_name'`. ```promql max by (nodeprefix) (ybnodeismasterleader{nodeprefix="$node_prefix"}) < 1 ``` Universe `'$universe_name'` database node has restarted `$value` times during last 30 minutes. ```promql max by (nodeprefix) (changes(nodeboottime{nodeprefix="$node_prefix"}[30m])) > 0 ``` Node to node CA certificate for universe `'$universe_name'` expires in `$value` days. ```promql min by (nodename) (ybphealthcheckn2ncacertvaliditydays{universe_uuid="$uuid"} < 30) ``` Universe `'$universe_name'` Master or T-Server has restarted `$value` times during last 30 minutes. ```promql max by (nodeprefix) (changes(ybnodeboottime{nodeprefix="$nodeprefix"}[30m]) and on (nodeprefix) (maxovertime(ybpuniverseupdateinprogress{nodeprefix="$node_prefix"}[31m]) == 0)) > 0 ``` More than one out of memory (OOM) kills have been detected for universe `'$universe_name'` on `$value` nodes. ```promql count by (nodeprefix) (ybnodeoomkills10min{nodeprefix="$node_prefix"} > 1) > 0 ``` `$value` database nodes are down for more than 15 minutes for universe `'$universe_name'`. ```promql count by (nodeprefix) (maxovertime(up{exporttype="nodeexport",nodeprefix="$node_prefix"}[15m]) < 1) > 0 ``` Node file descriptors usage for universe `'$universe_name'` is above 70% on `$value` nodes. ```promql count by (universeuuid) (ybphealthcheckusedfdpct{universe_uuid="$uuid"} > 70) ``` Last attempt to send alert notifications to channel `'{{ $labels.source_name }}'` has failed. You need to try sending a test alert to obtain details. ```promql lastovertime(ybpalertmanagerchannelstatus{customer_uuid = "$uuid"}[1d]) < 1 ``` Average node CPU usage for universe `'$universe_name'` is more than 90% on `$value` nodes. ```promql count by(nodeprefix) ((100 - (avg by (nodeprefix, instance) (avgovertime(irate(nodecpusecondstotal{job="node",mode="idle", nodeprefix="$node_prefix"}[1m])[30m:])) * 100)) > 90) ``` `$value` database Master or T-Server instances are down for more than 15 minutes for universe `'$universe_name'`. ```promql count by (nodeprefix) (labelreplace(maxovertime(up{exporttype=~"masterexport|tserverexport",nodeprefix="$nodeprefix"}[15m]), "exportedinstance", "$1", "instance", "(.*)") < 1 and on (nodeprefix, exporttype, exportedinstance) (minovertime(ybpuniversenodefunction{nodeprefix="$nodeprefix"}[15m]) == 1)) > 0 ``` `$value` nodes have inactive cronjob for universe `'$universe_name'`. ```promql ybpuniverseinactivecronnodes{universe_uuid = "$uuid"} > 0 ``` Node disk usage for universe `'$universe_name'` is more than 70% on `$value` nodes. ```promql count by (nodeprefix) (100 - (sum without (savedname) (nodefilesystemfreebytes{mountpoint=~"/mnt/.*", nodeprefix="$nodeprefix"}) / sum without (savedname) (nodefilesystemsizebytes{mountpoint=~"/mnt/.*", nodeprefix="$node_prefix"}) * 100) > 70) ``` Maximum clock skew for universe `'$universe_name'` is more than 500 milliseconds. The current value is `$value` milliseconds. ```promql max by (nodeprefix) (maxovertime(hybridclockskew{nodeprefix="$node_prefix"}[10m])) / 1000 > 500 ``` The tablet leader is missing for more than 5 minutes for `$value` tablets in universe `'$universe_name'`. ```promql max by (nodeprefix) (count by (nodeprefix, exportedinstance) (maxovertime(ybnodeleaderlesstablet{nodeprefix="$nodeprefix"}[5m])) > 0) ``` <!-- You can obtain information for alert templates from YugabyteDB Anywhere by executing the following command: ```sh aakra01@UCTDES01 ~ k exec yb-support-platform-yugaware-0 -c prometheus -n yb-platform -- ls /opt/yugabyte/prometheus/rules |more -5 yugaware.ad.034b2eec-23c6-4ef2-8a25-aab9820a4086.yml yugaware.ad.04b30133-e62d-44ef-a7c1-7aadb6bf4c39.yml yugaware.ad.0d28c7a4-e44f-4247-8343-029662075233.yml yugaware.ad.155697f8-f928-40a3-98e5-78f0b645c4f6.yml yugaware.ad.18175790-3fae-4008-83a8-41bb221a2cf7.yml --More-- aakra01@UCTDES01 ~ k get configmap/yb-support-platform-yugaware-prometheus-config -n yb-platform -o yaml |grep -i rule rule_files: - '/opt/yugabyte/prometheus/rules/yugaware.ad.*.yml' ``` -->