Tips and Tricks onusing useful features of Zabbix
in large scale environments

2023/10/06-07

NTT Com Engineering Corporation© NTT Com Engineering Corporation All Rights Reserved.

Self introduction

Engineering Corporation
AllAllRights
Reserved.
© NTTNTTComCommunications
CorporationRights
Reserved.

2

Who am I.
Joined company(2009)

Takashi Fukushima

I have been engaged in activities
related to Zabbix and monitoring
systems with Zabbix as the core forover 15 years,
including consulting, custom
development, support, and training.

from 2017from 2021

from 2023

Now© NTT Com Engineering Corporation All Rights Reserved.

3

Case study

Engineering Corporation
AllAllRights
Reserved.
© NTTNTTComCommunications
CorporationRightsReserved.

4

Company-A had the following requirements:
No.1
No.
01 Many branches (and systems)
No.2 Monitoring is centralized at the main office
No. 02-> Receive requests from System administrator of each branch
Branch 01

No.3
No.
03 There are variations in the system (each branch)

Branch 02

No.4
No.
04 Each branch have System administrator.
ZP

Main Office

ZP
Branch 03

Branch N
ZP
ZPZS© NTT Com Engineering Corporation All Rights Reserved.

5

The additional requirements.
Detected PROBLEM will be integrated with a ticket management system
Ex. 01 and managed within that system.

* Zabbix will only data collection and fault detection.
Web interface will not be used for reference.

Ex. 02 Targets servers and network devices.

The system should not only support automate registration but also automate updates.
Ex. 03 However, Rollback if update fails© NTT Com Engineering Corporation All Rights Reserved.

6

The additional requirements.

Ex. 04 Easily set up multiple log monitors and process monitors.

Ex. 05 Maintenance settings individually depending on the type of log message.

Automatically close PROBLEM events in log monitoring.
Ex. 06 (because the event specifications have changed since Zabbix 4.0)

© NTT Com Engineering Corporation All Rights Reserved.

7

This Zabbix has the following mechanism.

mechanism.1
Mechanism.
01 Automatic registration.
mechanism.2
Mechanism.
02 Automatic close PROBLEM events.
mechanism.3
Mechanism.
03 Flexible maintenance settings.

5.0

Zabbix API

CMDBAuto registrationtool© NTT Com Engineering Corporation All Rights Reserved.

Hosts:
Items:

600,000
1,400,000

Event
Auto closeAlertScriptsIssue Tracking system

8About the automaticregistration mechanism

Engineering Corporation
AllAllRightsReserved.
© NTTNTTComCommunications
CorporationRightsReserved.

9

About the automatic registration mechanism
The new registration of monitoring settings is carried out in the following 6 steps
START

4. Custom LLD

1. New registration
or Update of Host

5. Change
Host settings

2. Start
maintenance

6. Finish
maintenance

3. Enabling theHostComplete© NTT Com Engineering Corporation All Rights Reserved.

10About the automatic registration mechanism

Blue arrow indicate
important settings

STEP
STEP 01.
01 New registration or Update of HostCreate a host in Zabbix via Zabbix API
using host information extracted from CMDB via API.

Add Host tag

Add Prefix

ZSZabbix API

No proxyDisabled© NTT Com Engineering Corporation All Rights Reserved.

11

About the automatic registration mechanism
STEP 02
01. Start maintenance
Perform maintenance settings for the host registered in STEP 01.

With data collection

The host registered in Step 1
will be targeted.

Stop processing for 1 minute.
( Ensure that the maintenance status is active. )© NTT Com Engineering Corporation All Rights Reserved.

12About the automatic registration mechanism
STEP 03
01. Enabling the Host
Enable the host registered in STEP 01
and run config_cache_reload on the Zabbix server.

EnabledSet a wait time of 1-minute ( CacheUpdateFrequency seconds ) again.
© NTT Com Engineering Corporation All Rights Reserved.

13

About the automatic registration mechanism
STEP 04
01. Custom LLD

Send CSV data to the serverusing zabbix_sender.

Convert to JSON

© NTT Com Engineering Corporation All Rights Reserved.

14

About the automatic registration mechanism
STEP 04
01. Custom LLD

CSV to JSON&LLD macros

Various types of monitoring items can be easily generated.

© NTT Com Engineering Corporation All Rights Reserved.

15

About the automatic registration mechanism
STEP 05
01. Change Host settings

Change the “Monitored by proxy” settings on each host to the appropriate settings.

Modify© NTT Com Engineering Corporation All Rights Reserved.

16

About the automatic registration mechanism
STEP 06
01. Finish maintenance
The maintenance settings are removed using the Zabbix API, and monitoring is started.
Any hosts that experienced failures in the previous steps“newly registered hosts” are deleted as part of the rollback process.
The prefix is removed from newly registered hosts, reverting them to their appropriate names.

Delete Maintenance setting© NTT Com Engineering Corporation All Rights Reserved.

17

About the automatic close
PROBLEM events

Engineering CorporationAllAllRights
Reserved.
© NTTNTTComCommunications
CorporationRights
Reserved.

18

Two types of PROBLEM that Zabbix detects.

Type 1: PROBLEM and OK are always paired.

PROBLEM

OKFor example: Resource monitoring

We call this “stateful monitoring”.

© NTT Com Engineering Corporation All Rights Reserved.

19

Two types of PROBLEM that Zabbix detects.

Type 2: PROBLEM and OK do not necessarily pair.
PROBLEMOK？
For example: snmptrap, log monitoring

We call this “stateless monitoring”.

© NTT Com Engineering Corporation All Rights Reserved.

20

About automatic close PROBLEM events
Add a special tag to “Stateless monitoring”.

Special TagOmission

Be executed if PROBLEM-event has a special tag

Script uses Zabbix API’s “event.acknowledge” method and {EVENT.ID} macro to close the event.
© NTT Com Engineering Corporation All Rights Reserved.

21

About the flexible
maintenance settings

Engineering CorporationAllAllRights
Reserved.
© NTTNTTComCommunications
CorporationRights
Reserved.

22

About the flexible maintenance settingsSpecial Tag

Add the special tag "Message ID Number".

Omission

Special tags are defined by csv data to be set using LLD macros.

Special tags are defined for each type of log message detected.
© NTT Com Engineering Corporation All Rights Reserved.

23

About the flexible maintenance settings
Perform tag-maintenance settings for special tags.

Messages to unmonitor

Allows to perform maintenance of log monitoring on a message-type basis.
© NTT Com Engineering Corporation All Rights Reserved.

24

Everyone at Company-A was happy.

●●My Zabbix is always
monitoring over your servers.
Ha-ha!!

Employee of company A© NTT Com Engineering Corporation All Rights Reserved.

ZS
25

Troubleshootingcase 01

Engineering Corporation
AllAllRights
Reserved.
© NTTNTTComCommunications
CorporationRights
Reserved.

26

Troubleshooting example.

Please...
please help me!!
wowwwwwww

case 1

Error occurs
in automatic registration mechanism

ZS© NTT Com Engineering Corporation All Rights Reserved.

27

The cause of this trouble is very simple
Sent a very large amount of data.

Size of data part of csv : About 322KB

The problem occurred because within the program,zabbix_sender was being executed through the shell,
which Fails with "Argument list too long" error due to ARG_MAX limit

As a legendary trainer,
that's nothing special to me.

Trainer
10th Anniversary
Polo Shirt

Argument list limit© NTT Com Engineering Corporation All Rights Reserved.

28

There are other things to keep in mind.
Please read the Zabbix manual carefully.

https://www.zabbix.com/documentation/6.0/en/manual/discovery/low_level_discovery/notes#data-limits-for-return-values

UserParameters-> 512KB(up to Zabbix 5.0) or 16MB(Zabbix6.0 or later)When using Proxy-> 64KB(Using MySQL)

ZS© NTT Com Engineering Corporation All Rights Reserved.

ZP

64KB (Bottleneck)

512KB or 16MBZA
29

Troubleshootingcase 02

Engineering Corporation
AllAllRights
Reserved.
© NTT
NTTComCommunications
Corporation
RightsReserved.

30

© NTT Com Engineering Corporation All Rights Reserved.

31

Troubleshooting example.

Part 2

I was planning to enjoy my first vacation in a while.

© NTT Com Engineering Corporation All Rights Reserved.

32

Troubleshooting example.

Part 2

Zabbix server was down!!

Please...
please help me!!
Oooooops!!!

ZS
Mr. T.T.
( My team member )© NTT Com Engineering Corporation All Rights Reserved.

Company-A’s Zabbix server

33

2Whatproblemoccurred.
023/92/02293/」
290/2239/」
9/29」

NVPS decreased by 1/10.
Error message on Zabbix web interface.
Nodata function for hosts monitored directly from Zabbix Server is experiencing issues.

Finaly, the HistoryIndexCache became full and the Zabbix Server process was down.
Restarting the Zabbix Server and database did not resolve the issue.

© NTT Com Engineering Corporation All Rights Reserved.

34

What problem occurred.
These are the processes we focused on.
Configuration syncer
Task manager
Timer
Trapper
ipmp manager
icmp pinger
Unreachable poller(busy rate)
100%
90%
50%
95%
95%
95%
80%

We initially thought that the auto-close mechanism might be the cause.
Because process “task manager” seemed very busy. (near 100%)

Proposed to stop the mechanism of “Automatic close PROBLEM events”.
© NTT Com Engineering Corporation All Rights Reserved.

35

Propose temporary measures.
We proposed to stop the “automatic close PROBLEM events” mechanism.
The problem issue ended while Company-A was discussing whether to accept the proposal.
Zabbix resurrection!!
My Zabbix is always
monitoring over your servers.

●●ZS© NTT Com Engineering Corporation All Rights Reserved.

36

Case being settled
I enjoyed skiing.

ideal© NTT Com Engineering Corporation All Rights Reserved.

reality37

Problem occurred again.

Zabbix server was down again.

ZS© NTT Com Engineering Corporation All Rights Reserved.

38

New clues
Over 1,000 maintenance settings were in effect when the trouble occurred.
After disabling all maintenance settings, Zabbix system recovered.

This is a great clue.
Go home and read thesource code

© NTT Com Engineering Corporation All Rights Reserved.

39

but ......
My BOSS said.

Please read the source code now.
The only acceptable answers

for me are 'Yes' or 'OK.’

My BIG BOSS

© NTT Com Engineering Corporation All Rights Reserved.

40

Decipher source code on iPhone.

© NTT Com Engineering Corporation All Rights Reserved.

41

Found the specifications
Spec 01 HistorySyncer is checking if tags match.
Spec 02 Tagged maintenance needs to check whether the tag matches for every PROBLEM event.
- P ... Number of events generated.

Every time an event is generated, an M*N*O check is performed.
Spec 03- M ... Number of Tags attached to the event.
- N ... Number of Tags attached to the maintenance settings.
- O ... Number of active maintenance settings.
Therefore, the number of operations is the product of M * N * O * P.

* reference data ( Company-A ):- Sometimes, PROBLEM events were occurring in large numbers.
- Each event has 2 tags.
- Each maintenance setting has 2 tags.
but, a lot of tag-value (about 50)- Always, 500 over active maintenance settings. ( max:1500 over )© NTT Com Engineering Corporation All Rights Reserved.

42Permanent countermeasure plan.
Merge similar strings and reduce the total number of maintenance settings.
Reducing the number of target hostsMerge© NTT Com Engineering Corporation All Rights Reserved.

43

Things aren't going according to planned.

© NTT Com Engineering Corporation All Rights Reserved.

44

Zabbix is a very versatile and deep tool.
“ザビックスチョットデキル” trainer

Legendary trainer

RebornThe meaning of 'ザビックスチョットダケデキル' is 'I can use Zabbix a little. '© NTT Com Engineering Corporation All Rights Reserved.

45

Do you want this T-shirt ?

This T-shirt sold by a Zabbix Japan LLC

For inquiries, please contact ZabbixJapan staff.
© NTT Com Engineering Corporation All Rights Reserved.

46

Summary

Engineering Corporation
AllAllRightsReserved.
© NTTNTTComCommunications
CorporationRights
Reserved.

47

Summary
No.1 Custom LLD is a powerful and useful feature with great potential,
No.01

but it should be used with a good understanding of Zabbix's specifications.

No.2 Tag-level maintenance is a handy feature that allows
No.02
for flexible maintenance settings,

but excessive use tag-maintenance will increase the system load.

No.3 If you want to enjoy skiing, better not to bring your smartphone.
No.03© NTT Com Engineering Corporation All Rights Reserved.

48

T h a n k yo u f o r yo u r a t t e n t i o n .

© NTT Com Engineering Corporation All Rights Reserved.

49