Distributed latency monitoring
Anurag Bhatia, Hurricane ElectricAnurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Starts with idea of looking for smokeping
alternative...

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Smokeping●

Monitors latency, packet loss etc based on ICMP●

Supports ICMP, HTTP, DNS and many other “probes”●

Easy quick config●Can send email if high latency, packet loss etc is detected

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Challenges with Smokeping
1.

Hard to scale up

2.

Different locations need different setups / no easy federated view3.

No easy to club graphs based on source or destination
E.g 5 locations, 50 endpoints = 250 graphs!

4.

Limited alerting support

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Why look for alternative?
●

It’s important to watch out for latency between various endpoints●With many sources to many destinations, number of graphs can be very high& hence an aggregate view is important●It’s easier to have single tool to monitor latency, monitor servers, network
devices, application and API endpoints●Requirement to run setup in high availability design

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Prometheus
1.

Tool which in itself includes a tool to retrieve various metrics, store them aTime Series Database (TSDB), make them available over HTTP endpoint

2.

Works on a “pull model” by default where metrics can be pulled over fromendpoints which run “agent”

3.

Can store any metrics, with any set of labels like CPU, memory utilization,
storage utilisation, network interface traffic and even the latency!

4.

Prometheus server speaks to agent via HTTP(s) to pull these metrics at
predefined intervals
Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Prometheus design

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Everything is “metrics”...

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Everything is “metrics”...
●Metrics can be interpret / graphed in way needed●Possible to look at average (e.g 1 min average, 5 min average etc)●

One can attach various labels with metric (e.g dst_country: HK, dst_type:cloud etc)●

Support for alerting (via Alertmanager) based on predefined rule against ametricAnurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Example of metrics

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Introducing Blackbox exporter

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Blackbox exporter●Open source probing endpoint which can trigger measurement wheneverprobes●Probed over HTTP(s) endpoint with requirement arguments of host tomeasure●Supports HTTP, HTTPS, DNS, TCP, ICMP and gRPC

●Written in go, can be downloaded & executed as binary on server or as
docker container
Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

ICMP probe for “hknog.net” via Blackbox exporter

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

http_2xx probe for “hknog.net” via Blackbox exporter

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Prometheus Configuration examples...

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Prometheus query example

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Prometheus query example

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Prometheus query example

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Whatever can be queried, can be plotted...

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Grafana query

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Grafana query

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Whatever can be queried can be set to givealerts...

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Alerts setup

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Alerts setup

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Scaling up●Distribute monitoring endpoints
to various probes with logicallabels (country, region, type etc)

●Multiple prometheus servers forin hierarchical manner (supportfor federation)●Long term retention on S3
endpoints●Single alert manager running in
HA to de-duplicate●Support via Thanos, Cortex,Grafana mimir etc
Image source here

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

How this all fits in?

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

What about from distributed?

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

RIPE Atlas...

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

RIPE Atlas Exporter demo

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

RIPE Atlas Exporter config sample...

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Startlink -> My server in Nuremberg, GermanyAnurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

Summary●Metrics world is becoming standard for tooling●

Systems like Prometheus scale well. While initial learning curve is more thansingle integrated tools like Smokeping but these tools scale well●

There are many components which may or may not be used depending on
the need like federation, object storage offload etc.

●Labels play an important role for both graphing as well as alerting.

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India

References
1.

Prometheus (here)2.

Node exporter (here)

3.

Blackbox exporter (here)

4.

Alert Manager (here)

5.

Blog post Monitoring my home network by Karan Sharma (here)

6.

Replacing Smokeping with Prometheus (here)

7.

Smokeping_prober (here)

8.

Scaling up - Thaos, Cortex and Grafana mimir

9.

RIPE Atlas Exporter (here)

Anurag Bhatia - Hurricane Electric - Distributed latency monitoring - SANOG 41 - Mumbai, India