Migration to Zabbix 3.2 &Galera Cluster in large Bankingsector environment
Xavier SCHNEIDER
September 16th 2017

1. NATIXIS AT A GLANCESUMMARY

2. ZABBIX IN NATIXIS
3. MIGRATION METHODOLOGY TO ZABBIX 3.2 IN
LARGE ENVIRONMENT
4. GALERA ARCHITECTURE IN NATIXIS
5. GALERA RUNTIME EXPERIENCE
6. QUESTIONS2MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

13NATIXIS AT A GLANCEMIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT1NATIXIS AT A GLANCE
Natixis is part of BPCE Group, the second largest banking Group in France

4

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

1

NATIXIS AT A GLANCE

Natixis is the international corporate, investment, insurance and financial services arm of Groupe
BPCE! +16 000 employees! + 36 countries! € 8,7bn NET REVENUES

IT : Our expertise at the serviceof our clients

5

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT1NATIXIS IT AT A GLANCE
Natixis IT3 300 people
24/7 support

Cloud
From an infrastructure provider to a cloud broker

Attractiveprojects

Service Quality
Provide data powered proactive
service improvement
API enablement
Enable API development, deployment and management

DevopsBe an infrastructure resources broker
rather than a technical provider.

+ 7 200 physical hosts (+100 000 cores)+ 10 600 Vm hosts (+34000 cores)

6

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT27

ZABBIX IN NATIXISMIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT2ZABBIX IN NATIXIS

Number of hosts

20 324

Number of items

2 089 841

Number of triggers

1 538 461

Number of users

1 244

NVPS

7 500

Zabbix Server :2 Blade• 2 - INTEL(R) XEON(R) CPU E5-2620
V3 @ 2.40GHZ:X86_64(2394Mhz)• 128 Go RAM / 2 x 300Go HD• Linux : Redhat 7• Cluster VCS

Pairs dedicated•
DMZ! Tests web! Servers•
Hexadata Database•
Tests web8

14 Proxy servers :7 pairs• 2 - INTEL(R) XEON(R) CPU L5640 @2.27GHZ:X86_64(2266Mhz)• 20 Go RAM / 2 x 300Go HD• Linux : Redhat 6 & 7• Active/passive loadbalancing with F5

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENTFrontend
On 2 VMs• 2 Apache instances! Frontend Users! API2Zabbix in Natixis : Industrial processes
All tasks of affectation are automaticaly set :Users•Admin in Zabbix on their perimeter•Are responsibled about their own supervision•Declared automatically according to the outlook Email list associated with the production
teamHosts•

Automatically attached to Hostgroup associated with the production Team•System Template automatically link to the Host.

•Linked with System TeamAPI Frontend
About 100 jobs in Perl or Php

9MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT2Zabbix in Natixis
Toolbox Frontend :! Repetitive users actions! API calls

10

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT311MIGRATIONMETHODOLOGY TO
ZABBIX 3.2 IN LARGEENVIRONMENTMIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

3

Migration Methodology to Zabbix 3.2 in large environment

3.2.6

2.4.3

Target architecture
For Galera and partitioning, add columns and primary key

12

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

3

Migration Methodology to Zabbix 3.2 in large environment• Where are the limits ?
Stress module
Jan Garaj Project (https://github.com/monitoringartist/zabbix-server-stress-test)! 10 Zabbix Agents• Simulated loads of 4 000 hosts

30k
NVPS

Environment! Prod Database• restored on 2 nodes 43/44 (garbd on Zabbix Server),! All hosts are disabled except agents for Bench located in host group GALERA_BENCH.
! All actions are disabled. All proxies are configured as Active.
! Galera• mysql-wsrep-5.6.33-25.17-linux-x86_64• sst xtrabackup plugin.
• Process GARBD on Zabbix server! Zabbix 3.2.3

13

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

3

Migration Methodology to Zabbix 3.2 in large environment• Where are the limits ?
• Architecture limits

Zabbix Flow loadbalancedbetween Database! Dead locks

14

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENTMigration Methodology to Zabbix 3.2 in large environment

3• Where are the limits ?
•Stress limits• Tested till 30k Nvps (only numeric)without problem.
! This test generates only Mysql
inserts.

•15Tested at 20k Nvps
! Trigger with 1 condition created in
TPL System (4300 hosts)! Condition 8 added in sameTemplateo Row lock waits appear

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

3Migration Methodology to Zabbix 3.2 in large environment•

Which methodology to only have 5 minutes downtime ?

BEFORE D DAY - PREPARATIONS•••Archiving
Prepare all 3.2.6 binaries
Prepare new configs for new/obsolete parameters

utWitho dy anrotsihtrends

DRY RUNS
D DAY• Dump production (11 minutes)• Start Migration (while Zabbix 2.4 keeps running)
!
!
!
!
!
!

•Stop production 2.4 Zabbix Platform!
!

•Replace all binaries with appropriate symbolic links
Replace new Configuration files

Start Zabbix 3.2.6!
!

16

Restore Dump on new Database
Clean all orphan triggers inherited from 2.4.
Update to 3.2.6 on other Zabbix Server linked to new Database (3 minutes)
STOP other Zabbix server
Reset all triggers
Basic check from frontends

Check alerts
Check Zabbix Internal

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

Only 5minutes
downtime

417GALERA ARCHITECTURE
IN NATIXISMIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT4Galera architecture in NATIXIS

Dedicated server for backup• Extra Backup plugin• Stop database on thisnode easilyZabbix recommendations• All access to the same database

18Simple architecture••

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENTNo Database proxy
Switch config from Zabbix

519

GALERA RUNTIMEEXPERIENCEMIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENTGalera Runtime experience

5

SST Test on backup node!
!
!
!

Node 1 and 2 Synced
Stop node instance 3Delete all datas
StartTotal Time to execute : 1H42!
!
!
!

SST : 1h09
Prepare : 0h22
Move : 0h06
Start + resynchro : 0h05

20

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENTWith database in fullproduction (700 Go)

X

621

QUESTIONSMIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

722ANNEX

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT7AnnexAdd primary keys for Galera
Change keys for Galera:alter table dbversion ADD id4galera INT PRIMARY KEY AUTO_INCREMENT;
alter table history add id4galera int key auto_increment;
alter table history drop primary key , add primary key (id4galera,clock);alter table history_uint add id4galera int key auto_increment;
alter table history_uint drop primary key , add primary key (id4galera,clock);alter table history_log add id4galera int key auto_increment;
alter table history_log drop primary key , add primary key (id4galera,clock);alter table history_text add id4galera int key auto_increment;
alter table history_text drop primary key , add primary key (id4galera,clock);alter table history_str add id4galera int key auto_increment;
alter table history_str drop primary key , add primary key (id4galera,clock)23

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT

7

Annex
History* & trends* Tables Partitionning
Create procedure from http://zabbix.org/wiki/Docs/howto/mysql_partition
Use this for “partition_maintenance_all”:

24

MIGRATION TO ZABBIX 3.2 & GALERA CLUSTER IN LARGE BANKING SECTOR ENVIRONMENT