Golden rules and principles of our monitoring tools
By: Rafael Linares
Tags: Kpi, Software Development, Metrics, Monitoring, Edreams

Did you know that the eDO monitoring team has deployed over hundreds of influxdb, each one of which takes gigabytes of space on average, and that all of them send more than thousands of alerts per week?

Given the huge volumes of data that we, as a company, generate and manage, in the monitoring team, we have created some best practices that help teams maintain a healthy and reliable monitoring platform. But why should we keep it nice and tidy? Although the monitoring platform does not directly impact our customers’ experience nor our business, as usual, it does help us observe what is cracking everywhere, let’s say that we are like the Sauron eye, but nicer :)

So keeping a neat monitoring environment helps to:

Identify and measure performance problems and the stability of our systems.

Diminish the background noise when doing a KPI search, reducing MTTR in every team.

Efficiently use the resources of our infrastructure and have spare space for new relevant KPIs.

But, what is a relevant KPI made of?

To determine if a KPI has what it needs, the teams must agree on answering YES to the following two questions:

Is this data relevant to the company?

Does this data help us monitor a critical Service level indicator (SLI)?

If it’s a YES, then the KPI should enter our hall of fame and follow our guidelines for correct monitoring and handling, if not, it should be marked for disposal.

What types of KPIs do we have?

The Golden KPIs that are basically the standard ones, defined by the monitoring team and that set the foundation of our monitoring system e.g. volume of errors per minute, availability ratio, response time, etc.

that are basically the standard ones, defined by the monitoring team and that set the foundation of our monitoring system e.g. volume of errors per minute, availability ratio, response time, etc. The Custom KPIs that must be identified by the owner of an application and their team, and depends on the depth of knowledge on the corresponding topic.

And now that we know so much about it, let me illustrate it with a practical case.

Over the past few weeks, we had experienced issues storing daily data in an influxdb. We identified a module that was sending metrics which had more than 150,000 different possible values.

This event increased the resources consumption exponentially by occupying more than 60 gigabytes of RAM with not a request made to this data. If we try to add this data source to Grafana (for example), we get two possible results, the “not so bad” one, in which the query gets an answer that would take more than 5 minutes to populate the information, or the “too bad” one, in which this query takes too much RAM memory and causes the influxdb to crash. Both situations are unacceptable according to our service standards.

A solution was found by following these three simple steps:

We scheduled a meeting with the app team where we exposed the issue. The app team got together to identify and decide which were the application critical signals (SLA) and what metrics are really needed to be able to flag relevant behaviour anomalies over those signals. We then cleaned the influxdb from all the irrelevant metrics and alerts and applied the new metrics and rules which resulted in a substantial RAM consumption decrease from 70 gigabytes to 800 megabytes. As a result, queries in Grafana over that influxdb returned instant answers.

So, just to tie up all these ideas, the Cloud Technology that we, and most of the IT companies now use along with the continuously changing environment that we work in, increase the volatility and the volumes of the data that the platform generates and creates a need to adapt the monitoring continuously.

The capacity to store data can make a company fall on a false principle like “the more, the better” and this can cause a negative impact in cost, resources and space in the long run.

For these reasons, we may tend to over collect information and this could be a huge error. If we stop thinking by the day and get a more strategic approach, we may see that this behaviour could generate an unneeded cost and even the MTTR can be affected: more data, more noise to find what you are looking for.

Having stepped back at this stage of the integration, has allowed us to build a more strategic and healthier guideline to ensure we have a solid and efficient monitoring system.