www.clonera.net

Levent Karakaş

Real-time Streaming of Zabbix Metrics
to Big Data Platforms
Zabbix Summit 2020

Storing Metrics in RDBMS• Relational databases are not the ideal solution for storing
metrics at scale• Timescaledb support in Zabbix since version 4.2 is a huge relief(10x performance improvement)• What if there are terabytes of metric data to manage?

2

Who needs to manage
terabytes of metric data?
DevopsAlert• Best of breed monitoring tools,
all integrated

CloudPaging Apps
ITSM (Incedentmanagement)

Application• Zabbix is the main pillar• Multi tenant

IoT• Single-Sign-On (SSO)• Everything runs on containers• CI/CD Orchestration

OS & Virtualization
Dashboard

NetworkAutomation

Hardware3

ClonEYE Plarform{tenant}-monitoring-stackZabbix WebServer

Zabbix Postgres

MonitoringCluster

Zabbix Server

Tenant 1
Monitoring Stack
Other Application

Tenant 2
Monitoring Stack

Tenant 3
Monitoring Stack

Tenant n
Monitoring Stack

Another Application

Many containers in each stack

One stack for each tenant

4How to store lots of metrics?

• Cost effective• Cloud Native• Large Datasets• Stream Native• Replication• Batch Ingestion• Fault Tolerance• Time optimized partitioning• High Availability• Horizontal Scalibility• Scalability• Flexible Schema• High throughput• SQL Support

5How to import data?
• Batch Ingestion• Streaming• Bulk data import• Real-time• Supports many file formats and• Druid service ingests directly

file sourcesfrom streams• Task based

6

Zabbix Real-Time Export• Zabbix supports exporting to file (which can be batch digested)• File format is JSON (which is supported by Druid ingestion)• A new file is created when the file reaches to a size (Theyshould be moved, imported, deleted?)

https://www.zabbix.com/documentation/current/manual/appendix/install/real_time_export

7

Kafka for real-time streaming?
Zabbix ServerKafka

Druid Kafka
index service

DruidHadoop
HDFS• If we can stream zabbix events to kafka, Druid Kafka
index service can ingest• No files to manage!
8

Zabbix History Kafka Loadable Module
Zabbix Server

Zabbix History
Kafka ModuleKafka

Druid Kafka
index service

DruidHadoop
HDFS• Found a module in github• Tried writing a brand new one• Tested and realized that it is not• After lots of debugging, realized it isworking• Tried fixing

not possible to keep librdkafka state
open and reuse for events• Gave up
9Use a REST to kafka middleware
Zabbix ServerZabbix HistoryWebhook
ModuleKafka REST

Kafka

Druid Kafka
index service

DruidHadoop
HDFS• Kafka REST by Confluent
https://github.com/confluentinc/kafka-rest• Kafka Pixy by Mailgun

https://github.com/mailgun/kafka-pixy
10Zabbix History Webhook Module• Formats and writes Zabbix history to a webhook in zabbix
real-time export protocol in JSON (one additional tag: item key)
• Full support for float, integer, string, text and log• Content type can be specified (default: application/json)• SSL verification errors can be ignored

11

Zabbix History Webhook Module• Ability to call webhook per measurement or multiple
measurements (bulk mode)• Possibility to use custom tags while exporting in bulk mode(needed for kafka rest)• Environment variables supported (for ease of use with containers)• Can be used to export to any system that accepts web pushhttps://github.com/clonera/zabbix-history-webhook
12

Final Topology
Zabbix History
Kafka Module

ZabbixServerKafka REST

Kafka

Druid Kafka
index service

DruidHadoop
HDFS

Superset

Tenant 1
Files

Tenant 1
ZabbixTenant 1
Topic

Tenant 1
Task

Tenant 1
Data source

Tenant 1
Dashboards
Tenant 2
Files

Tenant 2
ZabbixTenant 2
Topic

Tenant 2
Task

Tenant 2
Data source
Tenant 2
Dashboards

Tenant nFiles

Tenant n
ZabbixTenant nTopic

Tenant n
Task

Tenant n
Data source

Tenant n
Dashboards

13Configuration #1: Zabbix Server
Environment variables to Zabbix Server Container
ZBX_LOADMODULE = "history_webhook.so"

ZBX_WEBHOOK_URL = "http://${kafka-rest}:38082/topics/${tenant}-zabbix"
ZBX_WEBHOOK_CONTENT_TYPE="application/vnd.kafka.json.v2+json"
ZBX_WEBHOOK_ENABLE_TEXT = "1"

ZBX_WEBHOOK_ENABLE_STRING = "1"(float and integer is enabled by default we also enable text and string here)
https://github.com/clonera/zabbix-history-webhook
14

Configuration #2: Kafka Rest
Environment variables to Kafka Rest Container
KAFKA_REST_ZOOKEEPER_CONNECT = "${zookeper}:2181"

KAFKA_REST_HOST_NAME = "${kafka-host}"
KAFKA_REST_LISTENERS = "http://0.0.0.0:38082"

https://docs.confluent.io/current/kafka-rest/quickstart.html
15

Configuration #3: Kafka
Environment variables to Kafka Container
KAFKA_BROKER_ID = 3

KAFKA_ZOOKEEPER_CONNECT = "${zookeper}:2181"
KAFKA_ADVERTISED_HOST_NAME = "${hostname}"
KAFKA_LISTENERS = "PLAINTEXT://0.0.0.0:9092"

KAFKA_LOG_RETENTION_HOURS = 48(9092 port is published on the host in this configuration)

16

Configuration #4: Druid HDFSDownload druid-hdfs-storage into extension folder and change
configuration file:# vi conf/druid/cluster/_common/common.runtime.properties
druid.extensions.loadList=["druid-hdfs-storage", "druid-kafka-indexing-service", "druid-datasketches","druid-influx-extensions", "postgresql-metadata-storage"]

druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://hadoop.local:8020/apps/druid/warehousehttps://druid.apache.org/docs/latest/development/extensions-core/hdfs.html
17Configuration #5: Druid Kafka indexing
Download druid-kafka-indexing-service into extensions folder andchange configuration file:# vi conf/druid/cluster/_common/common.runtime.properties
druid.extensions.loadList=["druid-hdfs-storage", "druid-kafka-indexing-service", "druid-datasketches","druid-influx-extensions", "postgresql-metadata-storage"]https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html
18Configuration #6: Druid ingestion

19

Configuration #6: Druid ingestion20

Configuration #6: Druid ingestion21

Configuration #6: Druid ingestion22

Configuration #6: Druid ingestion23

Configuration #6: Druid ingestion24

Configuration #7: Superset

25

ZabbixSuperset

26

Development Team

DevOPS Engineer

Software Developer

BI Expert

Software DeveloperDoğuş Peynirci

Burak KöseoğluRahma Bayhatun

Nurdan Kolay

27

Thanks