MONITORING EVOLUTION FOR
CLOUD NATIVE ENVIRONMENTS
Wang Manxue
Technical manager, China Mobile Online Service Co., Ltd, China

CONTENT010203

04

BackgroundWay outTransition

Precipitation

Centralized maintenancethroughout the country andthe biggest over the world

Choose open source

Several problems

Make the monitoring
have more possibilities

01
BACKGROUND——CENTRALIZED MAINTENANCE
THROUGHOUT THE COUNTRY AND
THE BIGGEST OVER THE WORLD

Respond to the challenge actively
Respond actively
Capacitybuilding

Quick

Centralized

AutomaticIntelligent

Provide the monitoring
capability in weekly
granularity“Chimney type monitoring”
divided by majormanually-added

Rely on the expertise

Provide the monitoring
capability in second level

Mixed and centralized
monitoring

Automatically closed loop
based on the strategy

Automatic identification
based on AI and big data

Huge amountof users••There are 900 million users and morethan 100 million WeChat fans. The
monthly service is more than 100million times; followers of Weibo are
30.38 million (be the first of theindustry); there are more than 50
million users for 10086 APP.
Ten thousand serversHighrequirement
Have high requirement andprovide telecom-level service•
99.999% reliability•
7* 24 guarantee

HardBefore

The business changes fast andthe operation environment iscomplicated•
Pop up online for 120
times daily and deal with500 work orders daily•Newtechnology:
Microservice/cloudcomputing/ container …

Now

ZABBIX SUMMIT ONLINE / 2020

operation challenge

02
WAYOUT——CHOOSE OPEN SOURCE• EMBRACE OPEN SOURCE• UNIFY THE MONITORING PLATFORM• BRIEF SUMMARYEmbrace open source

Non-real time: received CPU utilization
data which is 15 minutes ago
Single: do not support XX monitoring,
XX alerting and XX visualization
low synergetic effects : call up to knowwhether the business of XX department
have a alert ? Whether it recovers or not?

Monitoring requirement of new-period
monitoring fusion ofcross-domain, cross-level
and cross-manufacturer

Health inspection
Centralized monitoring
Unified alarm

Predicting fault
Customize spectaculars
Highly real-time

Fault location
Visualization

Resource control

Embrace open source and stand on the shoulder of giants-Zabbix
Rapid implementation: utilize the mature capacity of Zabbix and finish the capacity building of monitoring system rapidly in 1 month
Full coverage: the official templates of Zabbix and the templates from community can realize the monitoring coverage of multi-operation system andmainstream middleware rapidly
Real-time stability: Zibbix is very mature, and it can realize second-level data collection; there is no fault online in 3 years
Visualized dashboard: nice combination of Zabbix and Grafana , making abundant visualized dashboards rapidlyHigh efficiency and low cost: Zabbix's resource consumption is very low. The main consumption is that the databaseneeds physical hardware support, which takes less resources than PrometheusZABBIX SUMMIT ONLINE / 2020

Traditional monitoringUnify the monitoring platform: centralized construction,
unified control and standardized edge node
construction mode of centralized building, unified management, and standardized accessibility for all our branches.

Network centralization:◆The

headquarter

construction

oftakeschargemonitoringof

thecapability,standardization of the edge node and thecollection,analysis,expression

andnotification of all the monitoring data.
◆The branch provides resource maintain andmanage the monitoring resource by following

the standardized and packaged monitoring
templates.

ZABBIX SUMMIT ONLINE / 2020

In order to establish the monitoring ability more quickly and control the system quality more comprehensively, we adopted theSummaries: extensive, abundant, various and flexible
The dashboards can be made flexibly and the allocation can be finished in min-level. The diagram is shown in diversified: Linechart, bar chart, pie chart, area chart and topology,etc.

Network monitoring index

SNMP

DeviceLuminous
informatio
powernDevicestate

Networkcard state

Network
protocolSurvival/
packetloss/ timedelay

Port statePort
description

Sending
power ofopticalmodule

Link stateof BGP
peer pair

CPU/memory
occupancy

Broadband
utilizationof inletand outlet

SoftwareversionReceived
power ofopticalmodule

Ospf
neighborstate

Snmp state

Outlet/inlet
packageloss anderror
package

Systematicname

Temperatu
reInterface
type

State ofvrrp virtualrouter

ZABBIX SUMMIT ONLINE / 2020

Type and manufacture of network deviceSummaries: within half a year

2 million

900 000

300 000

Hosts

Monitoring
items

Triggers

Alerts

34%
84400

545

1.3

From 2014

Proxies+DashBoard

Lorem ipsum dolor sit amet,consectetur adipisicing elit, sed doeiusmod tempor incididunt ut labore
et dolore magnaUsersKActions

ZABBIX SUMMIT ONLINE / 202010000

Summaries: Zabbix System Optimization

CPU/memory/IOConnection(Maximum number of connections, timeout duration)
Data consistency

WEBNginx parameter
Php parameter
php.ini：max_input_vars（influence template application and a lot ofhosts fail）

Suggest adopting database SSD hard disk

ZabbixHost parameter
Kernel parameter
TCP protocol stack parameter
Semaphore/IO（Zabbix do not release signal set whenfails to start）

Configure the number of startup modules and processes
according to specific requirements
Forbid auto discovery, adopt script to call API

Forbid housekeeper and Enable database table partitioning
Forbid server connecting agent directly
Configuration parameter optimizaion
defines.inc.php：QUEUE_DETAIL_ITEM_COUNT（define the queue search
limitation of the monitoring item and influence the display of information queue ）ZABBIX SUMMIT ONLINE / 2020

DatabaseSummaries: dual-center, high-availability solution for Zabbix

Expose theunified domain
namekeepalived

Server1
hostServer2
cold standby

Center AMySQL
Headquarter and 31
provinces proxykeepalived

Server3

Server4

cold standby

cold standby

CenterBMySQL

Dual-center database DNS guarantees
the unified domain name and keep
pace with the configuration tablebesides history and trends table.

Headquarter and 31
provinces proxy

ZABBIX SUMMIT ONLINE / 2020

Deploy Zabbix system by A/B dual-center, provide service by exposing the unified domain name, and develop the capability that corresponding
proxy switch server with one-click. When the North-Center fault occurs, the server service can be switched by fast switch ZABBIX server domainname resolution. And switch all the proxy server by the automated scripts with the one-click . In case of fault occurs, it can ensure the quick recovery
of one key monitoring service within 10 minutes.

Experience1• Demand analysis andfunctional verification2• Promote and implement
in entire network

3• Performance optimization• Typical problems and software bugprocessing

4
Reworking means
efficiency reduced bymore than 3 times

Haste makes waste.
The quality can not be
guaranteed without
standardized delivery

1• Demand analysis andfunctional verification

2• Standard andspecification
formulation

3• Enactment and reformation ofstandardization

5• Interface and automation ofoperation and maintenance

Module,hostgroup,hostname, display name of
host, action name, content of
display board• Performance optimization• Typical problems and software bugprocessing

4
Requirement
delivery/change process, problem
handling process, regularmeeting and weekly report

5• Interface andautomation ofoperation andmaintenance

ZABBIX SUMMIT ONLINE / 2020

Suggestions(Standard is more than anything ,quality and efficiency equally important)

The History (Trial and Error)

03
TRANSITION——SEVERAL PROBLEMS
Q1: WE DIDN’T GET ANY NOTIFICATION WHEN OUR SERVICES WEREDOWN WHILE WE COLLECTING 2 MILLION METRICS?
Q2: IS THERE ANY VALUE IN USING MASSIVE LOGSBRIEF SUMMARY

Q1: we didn’t get any notification when our services were down while we collecting 2 million metrics?
Take the business quality and customer experience at the core and the target should be manageable, visualized and measurable.

Network centralized construction, centralized control and standardized access of edge node.

Software monitoring + hardware monitoring, unify, fusion and fluxion of operation data, construct multi-level measurement system.

Based on the user experience, establish full link monitoring from end to end; link alert, complaint pre-warning and customer service as whole closed—
loop system.

Operation guarantee
Capacity
Extension

Governancerange

Customer experiencemanagement

User problems management
User perception management

Business quality
management

Business problem management
Business quality management

Application systemmanagement

Facilities
management

Application performance management(PM)Application fault management(FM)Facility performance management(PM)Facility fault management(FM)Process andautomation
management

ZABBIX SUMMIT ONLINE / 2020Business and application quality is perceivable, which is the core of monitoring

Application performance management
Can find about 70% problems
for the monitor by applicationfrom the user experience

Sorting out thecore function
modules ofbusiness system

End user experience
Application program
Infrastructure

Sort out the coremonitoring
indicators offunctional modules

Extraction method andeffectiveness ofevaluation monitoring
indicators

Producemonitoring
dashboard

WE
B

MQ

JVM

DBOSServer

can find only about 30%
problems by the monitor of
infrastructure

Business quality monitoring

Refer to Google SRE 5 gold metrics
Application performance monitoring connects front-end with back-end service and user network environment toachieve end-to-end full link and code level monitoring.

User experienceratingFront-end
interactive
experience

Networkslicing

application
topology

Code location& tracking

1.
2.
3.
4.
5.

Rate: request rate, the number of requests per second.
Error: error rate, which is the number of errors per second.
Latency: response time, including queue / wait time, in milliseconds.
Saturation: the degree of overload, which is related to the resource utilization rate, andcan also be directly measured by queue depth.
Utilization: the busy level of a resource or system, usually expressed as 0% to 100%.

ZABBIX SUMMIT ONLINE / 2020

On the basis of strengthening the monitoring of basic settings, the application performance monitoring and service quality
monitoring capabilities are supplemented to ensure the business stability and customer perception.

Monitoring layer in cloud architecture

Contents that users care about

Contents that business site cares about

Monitoringcoverage

Business

Analysis of upstream and downstream,from end to end
Call and execution of service and method

LinkBasicmonitoring

ZABBIX SUMMIT ONLINE / 2020Businessmonitoring

User
experience

ApplicationSystemCloud platform
Power & environment monitoring

the problems
were found

Level of JVM and container
Level of OS

About 30% ofLevel of cloud platform

Container

About 70% ofLevel ofpower environment monitoring system

the problems
were found

Steadyoperation

Construction of an open, independent and self-controllable alerting monitoring monitoring system

SAAS
Monitoring spectaculars
Application ofmonitoring
Multi-index analysis
alerting scenarios
Volume management

D-PaaS
Operation andmaintenance data
analysis platform

Flink
Real-time dataprocessing

Operation dataanalysis

PaaS
Monitoring alertingbottom collection
platform

Log platform“1 platform, 4 systems and 3 capabilities”

Externalusers

Alarm query

Alarm statistics analysis

Root cause analysis

Fault statistics analysis

Fault self-recovery

Fault knowledge baseHadoopOff-line dataanalysis

CMDB
database

Operation dataanalysis

Enterprise dataresource

Prometheus

ZabbixAPM

automatic operation system

Regular data

machine learning
data
Technical network
management

Business
Applicat Databas middlew Contain Cloud PhysicalManagement
Internet
Journal monitori
er cloud platform device
areioneand control
ngresource object

Realhead

One platform- monitoring alerting platform
Four systems• Link system of four in one: control, monitor, manageand operation• Support system of decision analysis: handling the daily
faults basing on big data analysis and artificial
intelligence algorithm• Develop operation and cooperation system: constructhigh-efficient organization and cooperation system andopen operation capability• Evolution system of operation service: the operation
service can be measured with 360 degrees
3 capabilities• Automation capabilities: implement with highefficiency and improve the efficiency of personnel• Data capability: operating state is transparent and canoptimize digital drive system• Intelligent capability: utilize artificial intelligent
algorithm to assist operation to make decision and analyze

ZABBIX SUMMIT ONLINE / 2020

Internalusers

Realize automatic and intelligent operation scenarios from end to end

algorithm, workflow engine and fault tree to improve the efficiency and quality of operation work.

Improve the timelinessand accuracy of fault
detection

Reduce thetime for fault
notification

Reduce thetime for fault
requestReduce thetime for fault
location

Automatic
recommendation offeasible solutions

Fault levelrating

Fault
notification

Fault
requestFault
locationassistance

Handling
decision

Find Fault

Monitoring
systemAlerting
systemPlatform forprocess
management

Intelligent
analysis
systemDecision
systemReduce thetime for fault
switching

Fast
recovery offault

Fault
checking

Fault
rehearsing

Automatic operation
platform

ZABBIX SUMMIT ONLINE / 2020

Realize the automatic handling of fault operation from end to end through the basic capability construction of the platform and combining the rules,

04
PRECIPITATION——MAKE THE MONITORINGHAVE MORE POSSIBILITIES• AUTOMATIC ACCELERATION• DATA EMPOWERING

1. Automatic acceleration••••

Large enterprises have much basic resource, extensive business,frequent online change and huge monitoring configuration task
Monitoring addition can not be completed immediately and needs
frequent adjustment, and the repeated workload is large
The use barrier of open source tools is high. There is no userfriendly web page for most of it and it can be used flexibly aftertraining
The workers are all over the country; the basic resource has been
to 10 000 level; the business change is frequent; it is difficult forunified management

Solutions
1Standardization, streamlining and modularity ofmonitoring capability

2Second development, automation

3

Interface of configuration, Interface of data expression

At present, self-service monitoring can realize all kinds of selfservice monitoring, including adding, deleting, modifying andchecking, covering 85% of the company's monitoring needs. only
two engineers can maintain the whole monitoring system to chieve
followings:
1. Add the basic monitoring items of host resource application
process with one-click,
2. addition, deletion and modification of alerting contact,
3. query of monitoring coverage with one-click,4. regular inspection of Zabbix, etc.

ZABBIX SUMMIT ONLINE / 2020

Pain points

2. Data empowering
to analyze the data in multi dimensional.

Statement analysis•••

Analyze monitoring alerting
numbers according to thebusiness dimension, andoptimize the alerting
cooperatively.
Analyze request numbers,failure rate and other businessmetrics, reflecting the qualityof business operation.
Provide business healthweekly report for corebusiness

Application Extension• Fast fault location and root cause analysis

Enrich metrics•

storage, and add metrics like• Fault self-recovery: automatically enlarge andreduce volume, restart abnormally and interface
closing, etc.
• Resource management system realizes
volume management.

Break the barrier of datamonth-on-month, year-on-year.
•

Extend the period of

dashboard data.

ZABBIX SUMMIT ONLINE / 2020

Extract Zabbix, Prometheus, alerting platform, log platform, CMDB and other data to load on the big data analysis platform andCurrent figures

26 060 000

780 000

1 980 000

Hosts

Monitoring
items

Triggers

Alerting

975

84

Lorem ipsum dolor sit amet,consectetur adipisicing elit, sed doeiusmod tempor incididunt ut labore
et dolore magnaProxies

34%

60 billion
From 2014

Logs

1700+
DashBoardUsersZABBIX SUMMIT ONLINE / 2020

1 000 000

FutureZABBIX SUMMIT ONLINE / 2020

Depth
⚫ Anomaly detection of logs
⚫ Alerting compression &relevance⚫ Generation of alerts rules⚫ Volume management⚫ Performance management

Make intelligencelaunches in more
operation industries

Intelligent fault detecting

BreadthThank you!