Portal:Toolforge/Admin/Prometheus

This page contains administrator information about the Toolforge prometheus setup and how to manage it.
= Setup =
There should be a couple of VMs with the puppet role role::wmcs::toolforge::prometheus.
The VM instances should be big enough to hold all the metrics we collect (usually more than 100GB of data).
All the configuration (and metrics data) is stored in /srv/prometheus/tools/.
Among other things, this sets up a systemd service which is the main thing running in these server:
aborrero@tools-prometheus-03:~$ sudo systemctl status prometheus@tools.service
● prometheus@tools.service - prometheus server (instance tools)
 Loaded: loaded (/lib/systemd/system/prometheus@tools.service; static; vendor preset: enabled)
 Active: active (running) since Thu 2020-02-13 18:15:02 UTC; 15h ago
 Main PID: 1517 (prometheus)
 Tasks: 21 (limit: 4915)
 Memory: 9.4G
 CGroup: /system.slice/system-prometheus.slice/prometheus@tools.service
 └─1517 /usr/bin/prometheus --storage.tsdb.path /srv/prometheus/tools/metrics --web.listen-addr
[..]
Query targets are defined in profile::toolforge::prometheus in a very long inlined yaml.
NOTE: there is no relationship between these prometheus servers and cloudmetrics systems. They collect different metrics and use different grafana to show it.
== Redundancy ==
The HA approach is active/cold-standby. Both VMs collect exactly the same metrics, so there is no need for any specific sync between de VMs for data redundancy.
There is a web proxy with the name prometheus.svc.toolforge.org created using horizon, pointing to the active VM.
This proxy can be used to inspect the status of prometheus. Useful links:
* https://prometheus.svc.toolforge.org/tools/classic/targets
* https://prometheus.svc.toolforge.org/tools/graph
Worth noting that this URL is also what's used by our Grafana setup to use this Prometheus instance as a data source.
== Hiera ==
Some global hiera keys are needed for prometheus to be able to query for metrics:
prometheus_nodes:
- tools-prometheus-03.tools.eqiad1.wikimedia.cloud
- tools-prometheus-04.tools.eqiad1.wikimedia.cloud
Basically this sets up host firewall rules and other ACL mechanisms.
== Alerts ==
The Toolforge Prometheus hosts automatically provision their alert rules from the cloud/toolforge/alerts GitLab repository.
= Failover =
Since all prometheus VMs collect all the metrics, there is no need to do any specific sync before doing failover. Data should be already there in the backup/standby server.
* switch the proxy URL to point to the standby server, using Horizon.
* check that the proxy URL https://prometheus.svc.toolforge.org/tools/targets works when pointing to the new VM.
= Data migration =
In case you want to migrate metrics data, for example from an old VM to a new one, there are a couple of caveats:
* if you scp or rsync the data while prometheus is collecting them, the resulting data copy in the destination VM will be inconsistent and prometheus will refuse to start.
* to avoid consistency problems, the recommendation is to shutdown the prometheus@tools systemd service in both the source/destination VM while doing manual data syncronization.
= See also =
* Prometheus upstream docs on target configuration: https://prometheus.io/docs/prometheus/latest/configuration/configuration/
* Grafana instance to use: https://grafana.wmcloud.org