Monitoring Kafka on Kubernetes with Prometheus
By: Agraj Mangal
Tags: Grafana, Kafka, Monitoring, Prometheus, Kubernetes

Set it up !

Now that we have a basic understanding of what we want to achieve, let’s start with installing Prometheus & Grafana on our Kubernetes cluster. As always there are multiple options but we will stick with our friendly-neighborhood-operators to help us out with it. Let’s first create the namespace for our monitoring setup

kubectl create namespace monitoring

Step 1: Install the Prometheus Operator

kube-prometheus provides a quick way to deploy Prometheus operator, Prometheus server, Grafana, Highly available Alert Manager, Node Exporter, kube-state-metrics and other helpful tools. I personally feel like losing control when deploying everything under manifests for kube-prometheus so we will take a different way to deploy these components, but your mileage may vary so feel free to deploy everything using kube-prometheus. However if you want to deploy using kube-prometheus on AKS, you must label your nodes correctly. The deployment configuration of some of these manifests include the nodeSelector as

nodeSelector:

kubernetes.io/os: linux

This will not work by default with your AKS Cluster, so you must label your nodes accordingly using kubectl

kubectl label nodes <node-name> kubernetes.io/os=linux

What we’ll do instead for our example is to take things slowly and deploy things one-at-a-time, starting with Prometheus Operator: The Prometheus Operator requires certain Kubernetes RBAC related resources like ServiceAccount , ClusterRole and ClusterRoleBinding in order to be able to authenticate against the API server and watch the custom resources for Prometheus — all the code for this blog post can be found on Github so please clone the repository and follow along

Clone the repo from Github to follow along.

So let’s start by creating ServiceAccount, ClusterRole & ClusterRoleBinding for our Operator in the monitoring namespace

kubectl apply -f monitoring/prometheus-setup/prometheus-operator-service-account.yaml -n monitoring

kubectl apply -f monitoring/prometheus-setup/prometheus-operator-cluster-role.yaml -n monitoring

kubectl apply -f monitoring/prometheus-setup/prometheus-operator-cluster-role-binding.yaml -n monitoring

kubectl apply -f monitoring/prometheus-setup/prometheus-operator-deployment.yaml -n monitoring

Step 2: Create a Service Monitor

Before we go further with deploying Prometheus server, we need to configure Recording & Alerting rules; let’s must take a step back and understand a bit more about how the operator works, what are the custom resources it exposes, how they interact with each other etc.

Service Monitors are one of the Custom Resources offered by the Prometheus Operator and basically provides a way to tell the operator to target new Kubernetes Services for scraping by Prometheus. A ServiceMonitor should be deployed in the same namespace where Prometheus is deployed, although it could target services deployed in other namespaces also using namespaceSelector property. Here is the truncated version of the Service Monitor yaml used to scrape Kafka related metrics from the earlier deployed Kafka Cluster

Complete Code: Here

Couple of things to note here:

spec.selector.matchLabels is used to target the service by matching labels. spec.selector.namespaceSelector.matchNames is used to select the namespace in which the service is running.

kubectl apply -f monitoring/strimzi-service-monitor.yaml -n monitoring

Step 3: Setup some Prometheus Rules

Another Custom Resource PrometheusRule can be used to configure alert rules in Prometheus. Here is how one looks

Complete Code: here

kubectl apply -f prometheus-rules.yaml -n monitoring

Step 4: Edit the Kafka Cluster

Now is the time to edit the existing Kafka cluster ( we created in the earlier blog post ) and add the following items:

Step 4 a) Add Metrics for Kafka Brokers & Zookeepers

Complete code: here

Above is a snippet of how metrics are added to Kafka Brokers and Zookeeper. Prometheus offers four core metric types — Counter, Gauge, Histogram and Summary

Step 4 b) Kafka Exporter

While Prometheus has a JMX exporter that is configured to scrape and expose mBeans of a JMX target, Kafka Exporter is an open source project used to enhance monitoring of Apache Kafka brokers and clients by extracting additional metrics data from Kafka brokers related to offsets, consumer groups, consumer lag, and topics. Some useful examples of alerts that can be created based on these metrics are:

Under Replicated Partitions of a Topic

ConsumerGroup Lag is too large = Very Slow Consumers, falling behind producers

An alert can be created if there is no message on a topic for a configurable amount of time

Add the following snippet after spec.zookeeper to tls-kafka.yaml

Complete tls-kafka.yaml code here

Time to update our existing Kafka Cluster:

kubectl apply -f monitoring/tls-kafka.yaml

Kafka Exporter enhances monitoring of Apache Kafka brokers and clients by extracting additional metrics data from Kafka brokers related to offsets, consumer groups, consumer lag, and topics.

Step 5: Create the Prometheus Server

Now that we have the Operator running, we will create the custom resource for Prometheus specifying some config and the Operator will spin up a Deployment to manage the Prometheus server instance.

Complete Code — here

The above gist only shows the important piece of the puzzle — the Prometheus Custom Resource. The complete code is present here.

Before we deploy this, it is recommended to add some additional scraping targets for cAdvisor, kubelet, pods & services. They can be added to the Prometheus resource using spec.additionalScrapeConfigs property. To do so, we need to create the following Kubernetes Secret

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n monitoring

Now let’s deploy the Prometheus Server

kubectl apply -f monitoring/prometheus.yaml -n monitoring

And that’s it. The operator does the rest — creates the Prometheus server instance and a couple of services and we can now simply do a port-forward and open up the basic Web UI offered by Prometheus

kubectl port-forward prometheus-prometheus-0 9090:9090 -n monitoring

Now point your browser to http://localhost:9090 and you would see something similar to the below:

Local Prometheus Server

This UI can also be used to see configured Alerts, how many of them are active and and details about when they were fired.

Alerts UI

Details of Alert being Fired

We will see in subsequent sections on how to configure the Alert Manager to send out these alerts/notifications and how a much better visualization tool can be used to see these metrics in action !