High-cardinality TSDB benchmarks: VictoriaMetrics vs TimescaleDB vs InfluxDB
By: Aliaksandr Valialkin
Tags: Docker, Victoriametrics, Tsdb, Influxdb, Timescaledb

Intro

VictoriaMetrics, TimescaleDB and InfluxDB have been benchmarked in the previous article on a dataset with a billion datapoints belonging to 40K unique time series.

A few years ago there was Zabbix epoch. Each bare metal server had no more than a few metrics — CPU usage, RAM usage, Disk usage and Network usage. So metrics from thousands of servers could fit into 40K unique time series and Zabbix could use MySQL as a backend for time series data :)

Currently a single node_exporter with default configs exposes more than 500 metrics on an average host. There are many exporters exist for various databases, web servers, hardware systems, etc. All of them expose many useful metrics. More and more apps start exposing various metrics on itself. There is a Kubernetes with Clusters and Pods exposing many metrics. This leads to servers exposing thousands of unique metrics per host. So 40K unique time series is no longer a high cardinality. It becomes mainstream, which must be easily processed by any modern TSDB on a single-server.

What is a big number of unique time series at the moment? Probably, 400K or 4M? Or 40M? Let’s benchmark modern TSDBs with these numbers.

Benchmark setup

TSBS is a great benchmarking tool for TSDBs. It allows generating arbitrary number of metrics by passing the required number of time series divided by 10 to -scale flag (former -scale-var ). 10 is the number of measurements (metrics) generated per each scale host. The following datasets have been generated with TSBS for the benchmark:

400K unique time series, 60 seconds interval between data points, data covers full 3 days, ~1.7B total number of data points.

4M unique time series, 600 seconds interval, data covers full 3 days, ~1.7B total number of data points.

40M unique time series, 1 hour interval, data covers full 3 days, ~2.8B total number of data points.

Client and server were run on a dedicated n1-standard-16 instances in Google Cloud. These instances had the following configs:

vCPUs: 16

RAM: 60GB

Storage: 1TB standard HDD disk. It provides 120MB/s read/write throughput, 750 read operations per second and 1.5K write operations per second.

TSDBs were pulled from official docker images and were run in docker with the following configs:

VictoriaMetrics:

docker run -it --rm -v /mnt/disks/storage/vmetrics-data:/victoria-metrics-data -p 8080:8080 valyala/victoria-metrics

InfluxDB ( -e values are required for high cardinality support. See the docs for details):

docker run -it --rm -p 8086:8086 \

-e INFLUXDB_DATA_MAX_VALUES_PER_TAG=4000000 \

-e INFLUXDB_DATA_CACHE_MAX_MEMORY_SIZE=100g \

-e INFLUXDB_DATA_MAX_SERIES_PER_DATABASE=0 \

-v /mnt/disks/storage/influx-data:/var/lib/influxdb influxdb

TimescaleDB (the config has been adopted from this file):

MEM=`free -m | grep “Mem” | awk ‘{print $7}’` let “SHARED=$MEM/4”

let “CACHE=2*$MEM/3”

let “WORK=($MEM-$SHARED)/30”

let “MAINT=$MEM/16”

let “WAL=$MEM/16” docker run -it — rm -p 5432:5432 \

--shm-size=${SHARED}MB \

-v /mnt/disks/storage/timescaledb-data:/var/lib/postgresql/data \

timescale/timescaledb:latest-pg10 postgres \

-cmax_wal_size=${WAL}MB \

-clog_line_prefix=”%m [%p]: [%x] %u@%d” \

-clogging_collector=off \

-csynchronous_commit=off \

-cshared_buffers=${SHARED}MB \

-ceffective_cache_size=${CACHE}MB \

-cwork_mem=${WORK}MB \

-cmaintenance_work_mem=${MAINT}MB \

-cmax_files_per_process=100

Data loader was run with 16 concurrent threads.

This article contains only results for insert benchmarks. Select benchmark results will be published in a separate article.

400K unique time series

Let’s start with easy cardinality — 400K. Benchmark results:

VictoriaMetrics: 2.6M datapoints/sec; RAM usage: 3GB; final data size on disk: 965MB

InfluxDB: 1.2M datapoints/sec; RAM usage: 8.5GB; final data size on disk: 1.6GB

Timescale: 849K datapoints/sec; RAM usage: 2.5GB; final data size on disk: 50GB

As you can see from the results above, VictoriaMetrics wins in insert performance and in compression ratio. Timescale wins in RAM usage, but it uses a lot of disk space — 29 bytes per data point.

Below are CPU usage graphs for each of TSDBs during the benchmark: