Project: VictoriaMetrics (Observability and Analysis / Observability)
File: alerting-vmalert-managed-victoria-metrics.md

sort: 4 weight: 4 title: Alerting with vmalert and Managed VictoriaMetrics menu: docs: parent: "managed" weight: 4 aliases: /managed-victoriametrics/alerting-vmalert-managed-victoria-metrics.html This guide explains the different ways in which you can use vmalert in conjunction with Managed VictoriaMetrics <img src="alerting-vmalert-managed-victoria-metrics_setup.webp"> is installed. You can obtain it by building it from , downloading it from the , or using the for the container ecosystem (such as docker, k8s, etc.). is installed. You have a deployment in . If you are using helm, add the repository to your helm repositories. This step is optional. If you are using , make sure that it and its CRDs are installed. This step is also optional. You need to prepare file(s) with alerting or recording rules. An example file with one alerting rule. alerts.yml ```yaml groups: name: common rules: alert: instanceIsDown for: 1m expr: up == 0 labels: severity: critical annotations: summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up" description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute" ``` To use vmalert with Managed VictoriaMetrics, you must create a read/write token, or use an existing one. The token must have write access to ingest recording rules, ALERTS and ALERTSFORSTATE metrics, and read access for rules evaluation. For instructions on how to create tokens, please refer to this section of the . <img src="alerting-vmalert-managed-victoria-metricstokencreated_single.webp"> <img src="alerting-vmalert-managed-victoria-metricscopydatasource_single.webp"> <img src="alerting-vmalert-managed-victoria-metricstokencreated_cluster.webp"> <img src="alerting-vmalert-managed-victoria-metricscopyreadingdatasourcecluster.webp"> <img src="alerting-vmalert-managed-victoria-metricscopywritingdatasourcecluster.webp"> ```sh export TOKEN=81e8226e- export MANAGEDVMURL=https://gw-c15-1c.cloud.victoriametrics.com export ALERTMANAGER_URL=http://localhost:9093 ./vmalert -rule=alerts.yml -datasource.url=$MANAGEDVMURL -datasource.bearerToken=$TOKEN -notifier.url=$ALERTMANAGERURL -remoteWrite.url=$MANAGEDVMURL -remoteWrite.bearerToken=$TOKEN -remoteRead.url=$MANAGEDVM_URL -remoteRead.bearerToken=$TOKEN ``` ```sh export TOKEN=81e8226e- export MANAGEDVMURL=https://gw-c15-1c.cloud.victoriametrics.com export ALERTMANAGER_URL=http://alertmanager:9093 docker run -it -p 8080:8080 -v $(pwd)/alerts.yml:/etc/alerts/alerts.yml victoriametrics/vmalert:v1.87.1 -datasource.url=$MANAGEDVMURL -datasource.bearerToken=$TOKEN -remoteRead.url=$MANAGEDVMURL -remoteRead.bearerToken=$TOKEN -remoteWrite.url=$MANAGEDVMURL -remoteWrite.bearerToken=$TOKEN -notifier.url=$ALERTMANAGER_URL -rule="/etc/alerts/*.yml" ``` ```sh export TOKEN=81e8226e- export MANAGEDVMURL=https://gw-c15-1c.cloud.victoriametrics.com export ALERTMANAGER=http://alertmanager:9093 cat <<EOF | helm install vmalert vm/victoria-metrics-alert -f - server: datasource: url: $MANAGEDVMURL bearer: token: $TOKEN remote: write: url: $MANAGEDVMURL bearer: token: $TOKEN read: url: $MANAGEDVMURL bearer: token: $TOKEN notifier: alertmanager: url: $ALERTMANAGER config: alerts: groups: name: common rules: alert: instanceIsDown for: 1m expr: up == 0 labels: severity: critical annotations: summary: "{{ $labels.job }} instance:

}} is not up" description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute" EOF ``` ```sh export TOKEN=81e8226e- export MANAGEDVMURL=https://gw-c15-1c.cloud.victoriametrics.com export ALERTMANAGER=http://alertmanager:9093 cat << EOF | kubectl apply -f - apiVersion: operator.victoriametrics.com/v1beta1 kind: VMAlert metadata: name: vmalert-managed-vm spec: replicaCount: 1 datasource: url: $MANAGEDVMURL bearerTokenSecret: name: managed-token key: token remoteWrite: url: $MANAGEDVMURL bearerTokenSecret: name: managed-token key: token remoteRead: url: $MANAGEDVMURL bearerTokenSecret: name: managed-token key: token notifier: url: $ALERTMANAGER ruleSelector: matchLabels: type: managed apiVersion: v1 kind: Secret metadata: name: managed-token stringData: token: $TOKEN EOF ``` You can ingest metric that will raise an alert ```sh export TOKEN=81e8226e-* export MANAGEDVMURL=https://gw-c15-1c.cloud.victoriametrics.com/ curl -H "Authorization: Bearer $TOKEN" -X POST "$MANAGEDVMURLapi/v1/import/prometheus" -d 'up{job="vmalert-test", instance="localhost"} 0' ``` ```sh export TOKEN=76bc5470- export MANAGEDVMREAD_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ export MANAGEDVMWRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ export ALERTMANAGER_URL=http://localhost:9093 ./vmalert -rule=alerts.yml -datasource.url=$MANAGEDVMREADURL -datasource.bearerToken=$TOKEN -notifier.url=$ALERTMANAGERURL -remoteWrite.url=$MANAGEDVMWRITEURL -remoteWrite.bearerToken=$TOKEN -remoteRead.url=$MANAGEDVMREADURL -remoteRead.bearerToken=$TOKEN ``` ```sh export TOKEN=76bc5470- export MANAGEDVMREAD_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ export MANAGEDVMWRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ export ALERTMANAGER_URL=http://alertmanager:9093 docker run -it -p 8080:8080 -v $(pwd)/alerts.yml:/etc/alerts/alerts.yml victoriametrics/vmalert:v1.87.1 -datasource.url=$MANAGEDVMREADURL -datasource.bearerToken=$TOKEN -remoteRead.url=$MANAGEDVMREADURL -remoteRead.bearerToken=$TOKEN -remoteWrite.url=$MANAGEDVMWRITEURL -remoteWrite.bearerToken=$TOKEN -notifier.url=$ALERTMANAGERURL -rule="/etc/alerts/*.yml" ``` ```sh export TOKEN=76bc5470- export MANAGEDVMREAD_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ export MANAGEDVMWRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ export ALERTMANAGER=http://alertmanager:9093 cat <<EOF | helm install vmalert vm/victoria-metrics-alert -f - server: datasource: url: $MANAGEDVMREAD_URL bearer: token: $TOKEN remote: write: url: $MANAGEDVMWRITE_URL bearer: token: $TOKEN read: url: $MANAGEDVMREAD_URL bearer: token: $TOKEN notifier: alertmanager: url: $ALERTMANAGER config: alerts: groups: name: common rules: alert: instanceIsDown for: 1m expr: up == 0 labels: severity: critical annotations: summary: "{{ $labels.job }} instance: {{$labels.instance }} is not up" description: "Job {{ $labels.job }} instance: {{$labels.instance }} is not up for the last 1 minute" EOF ``` ```sh export TOKEN=76bc5470- export MANAGEDVMREAD_URL=https://gw-c15-1a.cloud.victoriametrics.com/select/0/prometheus/ export MANAGEDVMWRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ export ALERTMANAGER=http://alertmanager:9093 cat << EOF | kubectl apply -f - apiVersion: operator.victoriametrics.com/v1beta1 kind: VMAlert metadata: name: vmalert-managed-vm spec: replicaCount: 1 datasource: url: $MANAGEDVMREAD_URL bearerTokenSecret: name: managed-token key: token remoteWrite: url: $MANAGEDVMWRITE_URL bearerTokenSecret: name: managed-token key: token remoteRead: url: $MANAGEDVMREAD_URL bearerTokenSecret: name: managed-token key: token notifier: url: $ALERTMANAGER ruleSelector: matchLabels: type: managed apiVersion: v1 kind: Secret metadata: name: managed-token stringData: token: $TOKEN EOF ``` You can ingest metric that will raise an alert ```sh export TOKEN=76bc5470- export MANAGEDVMWRITE_URL=https://gw-c15-1a.cloud.victoriametrics.com/insert/0/prometheus/ curl -H "Authorization: Bearer $TOKEN" -X POST "$MANAGEDVMWRITE_URLapi/v1/import/prometheus" -d 'up{job="vmalert-test", instance="localhost"} 0' ```