Metrics Collection using Prometheus
By: Manjit Singh
Tags: Prometheus, Metrics Collection, Metrics, Node Exporter, Prometheus Exporters

In production systems, issues can happen at various levels. Based on metrics collection over a period of time, a good alerting/monitoring solution can help to avoid some of the issues or can help to take action before the issue affects users. Prometheus is one such solution that can be used independently for metrics collection and alerts or in conjunction with something like Grafana for better dashboards and alerts. Prometheus is a CNCF graduated project, so it has wide enterprise adoption and community support.

Prometheus mainly is a pull-based system that scrapes/collects from source systems/applications on set schedules. This part is taken care of by the scheduler component. Then metrics are stored in a time series database and exposed using API. Prometheus does have UI to run queries using its PromQL language. We can set alerts in Prometheus. Prometheus can get these metrics from windows and Linux servers using exporters or from defined metrics endpoints where applications publish their metrics to.

Scheduler scrapes metrics from servers, and applications

For batch processes, a push gateway can be used.

Exporters

Linux servers, windows servers, Kubernetes already collect a lot of metrics by default. By using exporters they can make those metrics available at standard endpoint like /metrics in Prometheus format. Prometheus can then collect these metrics at set schedules. These are in Key/value format but keys can have labels. For example, for Memory Available, it can be at just node level but for disk_io, it can be per disk by using label per disk.

There are a lot of exporters available for Databases, Hardware systems, Messaging systems, Storage systems, APIs, and so on. The list of exporters is available at the following link:

Exposing metrics from custom apps using Client Libraries

We can expose custom metrics from our applications like total jobs processed, currently executing requests, number of errors, and so on. There are client libraries for various languages (official and community) like GO, Python, Java, C#, NodeJS etc. By using these libraries metrics can be published to an endpoint in the required format that can be scraped by Prometheus on set schedules.

Counters and Gauges

Counters only record increasing values like total jobs processed, total errors encountered. Because all metrics are recorded with a timestamp in Prometheus, we can run various queries for rate, etc. for a given duration like a number of requests in the last 1 minute or a number of errors in the last 5 minutes. Once we have some established trends over a period of time, we can set some alerts when it is out of the expected range. For Counter metrics, the rate of change is far more valuable than absolute value.

Gauge values can go up or down like currently active jobs or memory used by applications. Normally it is the rate of change of counter combined with gauge that provides some insight. By comparing graphs of jobs being processed and memory consumed, we would expect to see the correlation between the two. We would expect memory usage to go up as jobs go up and come down as they finish processing but we know if memory is not released, there may be some underlying issue in the application.

Summary and Histogram metrics

Summary is used to record dependent values like how many files were processed in a given time and what was their total size. It is more about recording average. Then with multiple metrics recorded, we can get a rate of change of average.

Histogram records data in buckets. For example, we can record how many http requests took less than 100 milliseconds, less than 300 milliseconds, and so on. This gives us visibility into the distribution of data. There may be some outliers and they would be easy to spot.

Installing Prometheus

Prometheus can be installed as a stand-alone component in Linux/Windows or in Kubernetes. It can be downloaded from the following link for the requires OS and architecture:

https://prometheus.io/download/#prometheus

It’s very simple to set up. Download the zip/tar file, uncompress it and run it. This way it would just run with default config values as expected. Ideally, it would be set up as a service in Windows/Linux with some config changes. Two types of values can be updated. One is to set flags to run the Prometheus application. The second is to set up values for scrape targets etc.

Some of the common command-line flags that can be passed to start Prometheus are:

storage.tsdb.path — Where is data stored? By default, it is a data directory under the installation path. It can be set to some other standard data folder or on network share.

Where is data stored? By default, it is a data directory under the installation path. It can be set to some other standard data folder or on network share. config.file — Location of config file like /etc/prometheus/prometheus.yml

Location of config file like /etc/prometheus/prometheus.yml web.enable-lifecycle — If this is set to true, then we can make changes to the config file and then make HTTP request to refresh config without restarting Prometheus

If this is set to true, then we can make changes to the config file and then make HTTP request to refresh config without restarting Prometheus storage.tsdb.* — Values that would control how long/how much to retain data

Values that are set in the config file (set in config.file) affect what to scrape, when to scrape, from where to scrape:

scrape_interval

scrape_timeout

scrape_configs: defines jobs to scrape different targets

What happens when we start Prometheus:

When we start Prometheus, it would log some information to confirm a few things like what port it is listening on, database, and ready status. In my case, data storage was set to a location that triggered a warning.