Prometheus installation modes for k8s monitoring
By: José Hermosilla
Tags: Helm, Kubernetes, Monitoring, Prometheus

César González and José Hermosilla

Prometheus has become the de facto open source solution for Kubernetes monitoring, with multiple ways to install and run it. Those ways can be classified, firstly, attending to the locality of installation and then to the grade of automation and effort for the overall process.

For the external world, “cluster” refers to a production-ready Kubernetes deployment, and “locality” is the place of Prometheus’s components relative to the cluster, naming the possibilities as internal and external. Of course, a mix of both is possible, but without real advantages for more automated means of deployment.

The automation part, subject to the locality, includes multiple options, ranging from the need to build the infrastructure resources to run the monitoring components, to the definition of the desired configuration values for them. The process might require the launch of some virtual machines for running a high availability solution, or setup of a Helm Chart with the number of instances and the Docker image versions for a couple of Kubernetes’s StatefulSet.

What follows is a description and exploration of the above elements with the aim to help those that need to select an alternative to monitor Kubernetes. First, we present the constraint that locality imposes upon the installation modes. Then we describe each mode, continuing with some common elements to all modes, such as alert definitions and Grafana’s dashboards. At the end, a comparison summarises the principal elements of each alternative.

Installation modes

As mentioned before, there are multiple installation modes, from the launch of some virtual machines to provide a high available setup to a definition of a handful configuration values to specify the desired state of the final arrangement inside a Kubernetes cluster. These installation means are:

Custom installation

Prometheus Operator

kube-prometheus

Prometheus Operator Helm Chart

These options are constrained by the desired locality of the Prometheus components in relation to the cluster, with locality defined as the place of each component in regard to the cluster, internal or external. The following table summarizes that.

Installation modes and locality

Custom installation

Custom installation may be defined as the means by which a human operator assumes the burden of defining all the resources and configurations required to install and run the monitoring solution. Among other things, the steps include:

Select and launch virtual machines

Install and run Prometheus components: Prometheus, Alertmanager, Pushgateway, Node Exporter, cAdvisor

Install and run Grafana for visualizations

Setup for a high availability

Define the way to change and reload configurations (including Prometheus’s rules)

Upgrades of services and virtual machines

If the installation takes place outside Kubernetes, all these elements need to be manually defined; otherwise, only the second is needed, requiring the definition of Kubernetes resources with any of the multiple ways that currently exist: plain YAML files, Kustomize, jsonnet, Helm, etc.

But if the former is discarded, why carry the burden of defining what possibly exists? Guided by the Don’t Repeat Yourself principle, work repetition is unnecessary because the option is already present, leading to the next section.

Prometheus Operator

As the project itself established:

The Prometheus Operator provides Kubernetes native deployment and management of Prometheus and related monitoring components. The purpose of this project is to simplify and automate the configuration of a Prometheus-based monitoring stack for Kubernetes clusters.

With that goal, the Operator brings a set of custom resources and ensures that the current state matches the one specified by them. The custom resources are:

Prometheus

Alertmanager

ThanosRuler

ServiceMonitor

PodMonitor

PrometheusRule

The internal semantics and how it works can be found in the introduction post about the Operator, but for the present objective, what matters here is:

How to use it to deploy a production-ready monitoring solution? and

What elements does the Operator leave out?

The Github project has resources describing how to install the custom resources and examples of how to create the corresponding objects; but no enforcement regarding any tool to do that, aside from plain text YAML definitions. Based on the solution nature itself, it is obvious that the locality of the installation must be internal to the cluster.

On the other side the Operator does not prescribe the use of node-exporter, although it can be easily incorporated. The same applies to Grafana, without any set of predefined dashboards, and Prometheus rules are absent, too.

Of course all this manual work could be done one time and used thereafter. That is what the next mode offers.

The kube-prometheus way

If the Operator represents a more convenient way than a custom installation, kube-prometheus goes a step forward, using it as a starting point to build over and bring:

node-exporter and its related resources

Grafana with a ready-to-use set of dashboards

a predefined ensemble of Prometheus rules

A singular feature of this project is the language in which it is written, jsonnet, with the added possibility to use it as a library and combine with additional mixings or bundles, illustrated by the presence of the built-in set of rules and dashboards that result from the inclusion of the “Prometheus Monitoring Mixin for Kubernetes” project.

With a proper CI/CD pipeline, and after the initial configuration, a human operator only needs to add ServiceMonitor objects, rules and dashboards, within the same project or as part of an external bundle. This approach allows for self-contained projects that include the monitoring configuration alongside the source code. The pipeline outcome is a set of Kubernetes manifests generated by the library to be applied with kubectl.

As with the previous option, this solution is constrained to be inside the cluster, for the same reasons.

The project documentation is self-explanatory, with multiple examples to illustrate different configurations and arrangements.

Prometheus Operator Helm Chart

One of the most popular tools in the kubernetes ecosystem is Helm, a package manager for kubernetes applications. His hub offers a prometheus helm chart.

The chart will install the following components:

Prometheus-operator

Prometheus

Alertmanager

Node-exporter

Kube-state-metrics

Grafana

And also will create ServiceMonitor objects to scrape some internal kubernetes components like kube-apiserver, kube-scheduler, kube-controller-manager, etcd, kube-dns/coredns and kube-proxy.

It is worth noting, the workload controller that it is used depending of these components, meanwhile the Alertmanager and the Prometheus server are deployed as a StatefulSet, because they need data persistence, Grafana runs as a Deployment, and it is store the configuration and dashboards in ConfigMaps, the same happen with the operator. Finally the node exporter runs as a DaemonSet in all non master nodes of the cluster to be able to expose metrics for each one of them.

Some interesting parameter to change are:

The configuration directives and the alert template of the alert manager.

The labelling for the service monitor selector.

The way to access the components, by default the charts create a ClusterIP service that is only accessible inside the cluster, if you want to use an ingress to access the component it can be configured.

The storage of each component, by default does NOT create a persistence volume.

The credentials of each component.

The strongest points of this solution is the installation process and the simplification of the upgrade process, plus having all advantages described in the “Prometheus Operator” section.

As a constraint, this installation will only work for services monitored in the same cluster, having to configure some kind of federation like Thanos to solve this limitation.

Common things

Both kube-prometheus and the Prometheus Operator Helm Chart come with a default set of ready-to-use Grafana’s dashboards and rules; but a custom installation or a direct use of the Prometheus Operator imposes the need to add both of them. However, it is possible to take advantage of the “Prometheus Monitoring Mixin for Kubernetes” outcome and incorporate it into the solution workflow.

This way, although a certain amount of manual work is required for the initial setup, it’s possible to end with a custom solution in some manner similar to the more advanced ones.

Anyway, it is up to the human operator to select one of the above alternatives. The next and final section presents a summarized view of them.

Summary

Four modes were presented previously to accomplish the deployment of a monitoring solution for Kubernetes with the Prometheus ecosystem. Those modes form a continuity, covering the space from a custom setup with a considerable amount of manual effort to a more advanced one that only requires setting some configuration options to establish the desired end state.

The following table presents each of these setups with a comparison based on the locality with regards to the cluster and the amount of manual effort to get it up and running.

Installation modes summary

From the above is clear that kube-prometheus and “Prometheus Operator Helm Chart” are the more effortless solutions. But if the project constraints impose an installation which must be external to the cluster, then a custom option is the only way; in that case, at least the possibility of reuse of the “Prometheus Monitoring Mixin for Kubernetes” outcome is present.

Whatever the case, the option for a full stack monitoring solution for Kubernetes is there; the unthinkable is to not use it.

*Picture by Alina Grubnyak