Project: KubeBlocks by ApeCloud (App Definition and Development / Database)
File: alert.md

title: Configure alert description: How to enable alert keywords: [mysql, alert, alert message, email alert] sidebar_position: 2 Alerts are mainly used for daily error response to improve system availability. KubeBlocks has a built-in set of common alert rules and integrates multiple notification channels. The alert capability of KubeBlocks can meet the operation and maintenance requirements of production-level online clusters. :::note The alert function is the same for all. ::: The built-in generic alert rules of KubeBlocks meet the needs of various data products and provide an out-of-the-box experience without further configurations. These alert rules provide the best practice for cluster operation and maintenance, which further improve alert accuracy and reduce the probability of false negatives and false positives by experience-based smoothing windows, alert thresholds, alert levels, and alert indicators. Taking PostgreSQL as an example, the alert rules have built-in common abnormal events, such as instance down, instance restart, slow query, connection amount, deadlock, and cache hit rate. The following example shows PostgreSQL alert rules (refer to for syntax). When the amount of active connections exceeds 80% of the threshold and lasts for 2 minutes, Prometheus triggers a warning alert and sends it to AlertManager. ```bash alert: PostgreSQLTooManyConnections expr: | sum by (namespace,appkubernetesioinstance,pod) (pgstatactivitycount{datname!~"template.*|postgres"}) > on(namespace,appkubernetesio_instance,pod) (pgsettingsmaxconnections - pgsettingssuperuserreserved_connections) * 0.8 for: 2m labels: severity: warning annotations: summary: "PostgreSQL too many connections (> 80%)" description: "PostgreSQL has too many connections and the value is {{ $value }}. (instance: {{ $labels.pod }})" ``` You can view all the built-in alert rules in Alerts Tab of Prometheus Dashboards. Run the commands below to open Prometheus Dashboards. ```bash kbcli dashboard list kbcli dashboard open kubeblocks-prometheus-server # Here is an example and fill in the actual name based on the above dashboard list ``` The alert message notification of KubeBlocks mainly adopts the AlertManager native capability. After receiving the Prometheus alert, KubeBlocks performs steps including deduplication, grouping, silence, suppression, and routing, and finally sends it to the corresponding notification channel. AlertManager integrates a set of notification channels, such as Email and Slack. KubeBlocks extends new IM class notification channels with AlertManger Webhook. This tutorial takes configuring Feishu as the notification channel as an example. Before you start To receive alerts, you need to deploy monitoring add-ons and enable cluster monitoring first. Refer to for details. *Steps:* Configure alert channels. Refer to the following guides to configure your alert

* * :::note Each notification channel has its interface call volume and frequency limits and when the limits are reached, the channel will limit traffic and you cannot receive alerts. The SLA of the service provided by a single channel cannot guarantee the alerts are sent successfully. Therefore, it is recommended to configure multiple notification channels to ensure availability. ::: Configure the receiver. To improve ease of use, `kbcli` develops the `alert` subcommand to simplify the receiver configuration. You can set the notification channels and receivers by the `alert` subcommand. This subcommand also supports condition filters, such as cluster names and severity levels. After the configuration succeeds, it takes effect dynamically without the service restarting. Add an alert receiver. ```bash kbcli alert add-receiver --webhook='xxx' --cluster=xx --severity=xx ``` *Example* The following commands show how to add a receiver to Feishu based on different requirements. The webhook address below is an example and you need to replace it with the actual address before running the command. ```bash kbcli alert add-receiver \ --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo' kbcli alert add-receiver \ --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo,token=sign' kbcli alert add-receiver \ --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo' --cluster=mysql-cluster kbcli alert add-receiver \ --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo' --cluster=mysql-cluster --severity=critical ``` :::note For the detailed command description, run `kbcli alert add-receiver -h`. ::: View the receiver list to verify whether the new receiver is added. You can also view the notification configurations by this command. ```bash kbcli alert list-receivers ``` (Optional) Delete the notification channel and receiver if you want to disable the alert function. ```bash kbcli alert delete-receiver <receiver-name> ``` If you cannot receive alert notices, run the commands below to troubleshoot the logs of AlertManager and AlertManager-Webhook-Adaptor add-ons. ```bash kubectl get pods -n kb-system -l 'app=prometheus,component=alertmanager' kubectl logs <pod-name> -n kb-system -c prometheus-alertmanager kubectl get pods -n kb-system -l 'app.kubernetes.io/name=alertmanager-webhook-adaptor' kubectl logs <pod-name> -n kb-system -c alertmanager-webhook-adaptor ``` KubeBlocks also supports email alert. Configure SMTP server. ```bash kbcli alert config-smtpserver --smtp-from alert-test@apecloud.com \ --smtp-smarthost smtp.feishu.cn:587 \ --smtp-auth-username alert-test@apecloud.com \ --smtp-auth-password 123456abc \ --smtp-auth-identity alert-test@apecloud.com ``` View the SMTP server list to verify whether the above server is added successfully. You can also view the configuration details by this command. ```bash kbcli alert list-smtpserver ``` Add email receiver. ```bash kbcli alert add-receiver --email='user1@kubeblocks.io' ``` KubeBlocks email alert now supports receiving emails from a specified cluster and of a certain severity. You can set this function by using `--cluster` and `--severity` flags. `--cluster`: means only receiving emails from a specified cluster. ```bash kbcli alert add-receiver --email='user1@kubeblocks.io,user2@kubeblocks.io' --cluster=mycluster ``` `--severity`: means only receiving emails from a specified cluster and the alert severity is `warning`. ```bash kbcli alert add-receiver --email='user1@kubeblocks.io,user2@kubeblocks.io' --cluster=mycluster --severity=warning ```