Monitoring small HPC with Zabbix

Choose Your Weapon

Mark Vilensky
Scientific Computing Manager
Dept. of Chemical and Biological Physics
Weizmann Institute of ScienceRiga, October 2023

Monitoring small HPC with Zabbix

About usChemistry Faculty• Five departments• 14 heavy usage groups, ~50 active users• Long HPC history (clusters, Altix)• Faculty cluster was planned in 2009 and built in 2010, and it is continuously growing

2

Monitoring small HPC with Zabbix

Cluster topology• 584 dual socket compute nodes• Memory ranges between 4 to 16 GB/core• Total 25,652 cores• Two Infiniband partitions 5:1 blocking and non-blocking• GPFS/DDN• CentOS 7.9• PBS Professional• Management servers on VMware, ~25 VMs• Staff = 3!

3

Monitoring small HPC with ZabbixHardware• Compute:• HPE Apollo 2000 gen 10• HPE ProLiant DL360 gen 9• Dell C6420• Dell C6520• Dell R750
• Login nodes• Infiniband switches: EDR, HDR• Infiniband adapters: EDR, HDR/100• Ethernet switches – HPE 5406R zl2, Aruba 2930M, Aruba
6300M• Storage – DDN 7990 embedded• Head nodes – VMware on 2 x Lenovo SR630 V2, Lenovo
DE4000H

4

Monitoring small HPC with Zabbix

HistoryProblems• Operating system level• Nagios• Passive checks• Ganglia• Hardware• HP SIM• HPE OneView• Dell OME / Cloud IQ• HPE switches• Nagios R.I.P• Ganglia R.I.P since 2018
• Python 2• HPE switches• Useless alerts• Flooding alerts• Complicated setup• Unexpected faults

5

Monitoring small HPC with Zabbix

Requirements

Alternatives• SCALABILITY!
• Auto discovery• Host groups• Custom checks• Only useful alerts• Smooth operation• Picture for the management• Zabbix• Prometheus• Commercial?

6

Monitoring small HPC with Zabbix

A secret link https://www.youtube.com/@DmitryLambert

7

Monitoring small HPC with Zabbix

Zabbix server• Version: 6.0.19• 16 vCPU• 32 GB RAM• 105 GB disk for OS• 200 GB disk for data
• 3 NIC• Rocky 8.7• PostgreSQL 14.6.1• Timescale DB 2.8.1• History – 14 days 180 days
• Trends – 60 days 1 year8

Monitoring small HPC with Zabbix

Zabbix server - vCenter view (one month)9

Monitoring small HPC with Zabbix

Zabbix server - vCenter view (one month)10

Monitoring small HPC with Zabbix

Zabbix server - vCenter view (one month)
Disk

11

Monitoring small HPC with Zabbix

How many metrics?

12

Monitoring small HPC with Zabbix

Main Dashboard

13

Monitoring small HPC with Zabbix

External scripts and custom checks and actions• PBS• Offline nodes with comment are set to “maintenance”• GPFS• Node health• Connectivity• Waiters• FlexLM vendor and daemon status on the license servers• Specific services on all infrastructure servers• Repair Ethernet module on some compute nodes

14

Monitoring small HPC with Zabbix

Huston! Do we have a problem?

15

Monitoring small HPC with Zabbix

Host groups• Compute• CPU• GPU• Ethernet switches• Infrastructure:• Login nodes• PBS• GPFS, e.g. NSD• Name and licenses servers• Provisioning (xCAT)• Monitoring servers

16

Monitoring small HPC with Zabbix

Better call admin
Saul - notifications• Why only “high” and “disaster”?
• Difference between compute and infrastructure:• Compute – alive or not, maintenance via PBS17Monitoring small HPC with Zabbix

A picture is worth a thousand words or, maybe $$$ …

19

Monitoring small HPC with Zabbix

Conclusions and Thanks• Metrics outside Zabbix:• GPFS GUI• BeeGFS• UFM – Infiniband metrics• Minor upgrade - smooth• RHEL?
• Only two dashboards• Not using reports• Special thanks to Dr. Andrei Vasilev• Questions

21

Monitoring small HPC with Zabbix

Thankyou.