Monitoring a server cluster using Grafana and InfluxDB
By: Antoine Solnichkin
Tags: Dashboard, Software Development, DevOps, Data Visualization, Programming

When a HTTP request is received, the load balancer will proxy (i.e forward) the request to the appropriate node to ensure that the load is equally divided among the cluster. Load balancers use different kind of techniques to decide which node to send to, but in our case, we will use an unweighted Round Robin configuration : a request will be sent to the first server, then the second and so on. No preference will be made regarding the node to choose.

Now that we have defined all the technical terms, we can start to implement our monitoring system.

II — Choosing The Right Stack

For this project, I will be using a Xubuntu 18.04 machine with a standard kernel.

In order to monitor our server cluster, we need to choose the right tools. For the real-time visualization, we are going to use Grafana v6.1.1

For monitoring, I have chosen InfluxDB 1.7 as a datasource for Grafana as it represents a reliable datasource for Grafana. In order to get metrics for InfluxDB, I have chosen Telegraf — the plugin-server driven agent created by InfluxData. This tutorial does not cover the installation of the tools presented ahead, as their respective documentation explains it well enough.

Note : make sure that you are using the same versions as the ones used in this tutorial. Such tools are prone to frequent changes and may alter the validity of this article.

III — Setting Up A Simple HA Cluster

In order to monitor our HA cluster, we are going to build a simple version of it using NGINX v1.14.0 and 3 Node HTTP server instances. As shown in the diagram above, NGINX will be configured as a load balancer, proxying the requests to our Node instances.

If you already have a HA cluster setup on your infrastructure, feel free to skip this part.

a — Setting NGINX as a load balancer

NGINX is configured to run on port 80, running the default configuration, and proxying requests to services located on port 5000, 5001 and 5002.

NGINX configured as a simple load balancer

Note the /nginx_status part of the server configuration. It is important not to miss it as it will be used by Telegraf to retrieve NGINX metrics later.

b — Setting simple Node HTTP instances.

For this part, I used a very simple Node HTTP instance, using the http and httpdispatcher library provided by Node natively.

A simple HTTP server written in Node

This server does not provide any special capabilities but it will be used as a basic web server for NGINX to proxy requests to.

In order to launch three instances of those web servers, I am using pm2 : the process manager utility for Node instances on Linux systems.

Now that NGINX is up and ready, let’s launch our three instances by running :

Doing this to the two other instances of Node servers, we have a cluster of three Node nodes up and ready.

Our three node are up and running!

IV — Setting Up Telegraf For Monitoring

Now that our HA cluster is built and running, we need to setup Telegraf to bind to the different components of our architecture.

Telegraf will be monitoring our cluster using two different plugins :

NGINX plugin : used to retrieve metrics for NGINX servers such as the number of requests, as well as the waiting / active or handled requests on our load balancing server.

: used to retrieve metrics for NGINX servers such as the number of requests, as well as the / or requests on our load balancing server. HTTP_Response : used to periodically retrieve the response time of each node, as well as the HTTP code associated with the request. This plugin will be very useful for us to monitor peaks on our nodes as well as node crashes that may happen.

Before starting, make sure that telegraf is running with the following command : sudo systemctl status telegraf . If your service is marked as Active , you are ready to go!

Head to Telegraf default location for configuration ( /etc/telegraf ), edit the telegraf.conf file and add the following output configurations to it.

Configuration for the NGINX plugin of Telegraf

Configuration for each individual node

When you’re done with modying the configuration of Telegraf, make sure to restart the service for the modifications to be taken into account. ( sudo systemctl restart telegraf ).

Once Telegraf is running, it should start sending periodically metrics to InfluxDB (running on port 8086) in the telegraf database, creating a metric called by the name of the plugin running it. (so either nginx or http_response ).

If such databases and measurements are not created on your InfluxDB instance, make sure that you don’t have any configuration problems and that the telegraf service is correctly running on the machine.

Now that our different tools are running, let’s have a look at our final architecture before jumping to Grafana.