Prometheus vs VictoriaMetrics benchmark on node_exporter metrics
By: Aliaksandr Valialkin
Tags: Victoriametrics, Benchmark, Monitoring, Prometheus

Recently single-node VictoriaMetrics gained support for scraping Prometheus targets. This made possible to run apples-to-apples benchmark, which compares resource usage for Prometheus and VictoriaMetrics during scraping big number of real node_exporter targets.

Benchmark setup

The benchmark was run in Google Compute Engine on four machines (instances):

An instance with node_exporter v1.0.1 for scraping . It was run on e2-standard-4 machine with the following config: 4vCPU, 16GB RAM, 1TB HDD persistent disk. Initial tests revealed that the node_exporter cannot process more than a few hundred requests per second. Prometheus and VictoriaMetrics were generating much higher load on the node_exporter during tests. So it has been decided to put nginx in front of node_exporter with one-second response caching. This lowered load on node_exporter to reasonable values, so it could process all the incoming requests without scrape errors.

. It was run on e2-standard-4 machine with the following config: 4vCPU, 16GB RAM, 1TB HDD persistent disk. Initial tests revealed that the cannot process more than a few hundred requests per second. Prometheus and VictoriaMetrics were generating much higher load on the during tests. So it has been decided to put in front of with one-second response caching. This lowered load on to reasonable values, so it could process all the incoming requests without scrape errors. Two dedicated e2-highmem-4 instances for Prometheus v2.22.2 and VictoriaMetrics v1.47.0 with the following configs: 4vCPU, 32GB RAM, 1TB HDD persistent disk. Both VictoriaMetrics and Prometheus were run with default configs except of the path to the file with scrape configs (i.e. -promscrape.config=prometheus.yml for VictoriaMetrics and — config.file=prometheus.yml for Prometheus). The prometheus.yml file has been generated from the following Jinja2 template:

global:

scrape_interval: 10s

scrape_configs:

- job_name: node_exporter

static_configs:

{% for n in range(3400) %}

- targets: ['host-node-{{n}}:9100']

labels:

host_number: cfg_{{n}}

role: node-exporter

env: prod

{% endfor %}

All the host-node-{{n}} hostnames were pointed to the machine with node_exporter target. They were put in /etc/hosts . This emulated scraping from 3400 node_exporter targets.

An e2-standard-2 machine for monitoring VictoriaMetrics and Prometheus. VictoriaMetrics instance on this machine has been configured for scraping app-specific metrics and node_exporter metrics from machines with VictoriaMetrics and Prometheus. Graphs below were built from these metrics.

The node_exporter has been chosen as scrape target because of the following reasons:

node_exporter is the most widespread exporter, which is scraped by the majority of Prometheus installations.

is the most widespread exporter, which is scraped by the majority of Prometheus installations. node_exporter exports real-world metrics (CPU usage, RAM usage, disk IO usage, network usage, etc.) under load, so benchmark results could be extrapolated to production Prometheus setups.

Ubuntu 20.04 has been installed on all the machines mentioned above.

Both VictoriaMetrics and Prometheus were started at the same time for scraping the same target node_exporter . The benchmark duration was 24 hours.

Storage stats

Let’s look at storage stats, which is the same for both VictoriaMetrics and Prometheus:

Ingestion rate: 280K samples/sec

Active time series: 2.8 million

Samples scraped and stored: 24.5 billion

Benchmark results

Disk space usage: