Kubernetes Monitoring with Prometheus & Grafana | Towards Data Science

In Part I and Part II of the Practical Monitoring with Prometheus and Grafana series, we installed the Prometheus blackbox exporter to probe HTTP endpoints and deployed our monitoring stack to Kubernetes via Helm. In this post, we will complement our black-box monitor with white-box monitoring techniques, namely anomaly detection using z-scores.
While there are plenty of tutorials on monitoring Kubernetes metrics or using the Prometheus client library to scrape application data, few posts cover anomaly detection techniques on common metrics such as HTTP requests and message queue size. This post will walk through an example scenario involving a web server communicating to other microservices via a message queue. We will attempt to detect anomalies in HTTP requests and a spike in message queue size to create smart alerting rules.
NOTE: For preconfigured Prometheus metrics and alerting rules, you can use Awesome Prometheus Alerts or modify default alerts from Prometheus Operator. If you have Thanos set up and want to dive deeper into ML models, you can also look at AlCoE’s Prometheus Anomaly Detector project.
To run anomaly detection, we’ll first need data. I’ll be using a NodeJS/Express server as an example, but you can find the corresponding Prometheus client for your language of choice to follow along.
npm i express-prometheus-middleware
We will just use default settings (i.e. expose Prometheus metrics to the /metrics endpoint and collect default NodeJS metrics) and specify HTTP request duration buckets.
const express = require('express')const promMid = require('express-prometheus-middleware')const app = express();// ... other express setup hereapp.use(promMid({  metricsPath: '/metrics',  collectDefaultMetrics: true,  requestDurationBuckets: [0.1, 0.5, 1, 5, 10]}));app.listen(8080)
Now the server will collect http_requests_total (a counter for total requests received with route, method, and status labels) and http_request_duration_seconds (HTTP duration in seconds with route, method, and status labels). Make a Docker image, add the Prometheus scrape annotations, and deploy the Helm chart on Kubernetes.
apiVersion: apps/v1kind: Deploymentmetadata:  name: {{ .Chart.Name }}  labels:    app: {{ .Chart.Name }}spec:  selector:    matchLabels:      app: {{ .Chart.Name }}  template:    metadata:      labels:        app: {{ .Chart.Name }}      annotations:        prometheus.io/port: "8080"         prometheus.io/scrape: 'true'// ... rest of deployment.yaml
For the message queue, I’ll be using Google Pub/Sub and monitor via Stackdriver, but you can replace it with Prometheus exporters for Kafka, RabbitMQ, or other queue implementation of your choice.
To deploy the Stackdriver exporter, first create a stackdriver.yaml file to specify GCP project ID and Stackdriver metrics to monitor. I’m using a few PubSub metrics, but you can also find the full list of available GCP (and AWS) metrics here.
stackdriver:  projectId: "<my-gcp-projectId>"  metrics:    typePrefixes: 'pubsub.googleapis.com/subscription/num_undelivered_messages,pubsub.googleapis.com/subscription/streaming_pull_ack_request_count,pubsub.googleapis.com/topic/message_sizes'annotations:  prometheus.io/port: "9255"  prometheus.io/scrape: "true"
Now deploy the chart and wait until metrics start to show up:
helm install -f stackdriver.yaml stackdriver-exporter stable/stackdriver-exporter// or Helm 2// helm upgrade --install stackdriver-exporter stable/stackdriver-exporter -f stackdriver.yaml
“To detect anomalies perfectly, at the right time, is not possible”- John Allspaw from An Open Letter To Monitoring/Metrics/Alerting Companies
The tricky part of performing anomaly detection on time-series data using Prometheus is setting reasonable aggregations to the data. For our HTTP requests and PubSub metrics example, we have two choices to make:
What labels do we use to group the data? Should it be by service, route, or environment?What time period do we use to determine rate values? 1m, 5m, 10m?
What labels do we use to group the data? Should it be by service, route, or environment?
What time period do we use to determine rate values? 1m, 5m, 10m?
If the aggregation spans too much data, anomalies within subsets of our dataset may be obfuscated by overall trends. On the other hand, if the aggregation is too granular, then our model is susceptible to false positives. The level of aggregation will depend on your use case, but in most cases, using services (i.e. server for HTTP requests and topic/subscription for Pub/Sub) in 5m intervals seemed to be optimal.
Advanced time-series anomaly detection techniques such as state-space models (ARIMA, Holt-Winters), decomposition (STL), or dimensionality reduction (RPCA, SOM) are impractical to do using Prometheus exclusively. To try those popular models, you can use Thanos to export your data and use R (Twitter’s AnomalyDetection) or Python (LinkedIn’s Luminol). But for us, assuming that our traffic is mostly normally-distributed, simple z-scores will get us started.
A z-score is a statistical measure determined by the number of standard deviations(σ) the raw score is from the mean. A z-score of 0 would mean that our score matches the mean of our data set. Using a standard normal distribution, we would then know that 95% of the data falls within ~2σ from the mean and ~99.7% of the data falls within 3σ.
Now, assuming a normal distribution for our HTTP request counts and PubSub metrics, we can use the z-score to flag data outside 3σ as anomalies and trigger alerts.
With aggregation levels defined, we need to create recording rules to save precomputed values for faster z-score analysis. For example, we can create a metric that captures the HTTP request rate in 5m intervals, grouped by app like:
- record: job:http_requests:rate5m  expr: sum by (app) (rate(http_requests_total[5m]))
Likewise, for PubSub metrics, we can track message size in 5m intervals grouped by topic:
- record: job:pubsub_message_size:rate5m  expr: sum by (topic_id) (rate (stackdriver_pubsub_topic_pubsub_googleapis_com_topic_message_sizes_count[5m]))
To apply the recording rules, append the recording_rules.yml section under serverFilesin the Helm chart:
serverFiles:  recording_rules.yml:    groups:      - name: PubSubRules        rules:          - record: job:pubsub_message_size:rate5m            expr: sum by (topic_id) (rate(stackdriver_pubsub_topic_pubsub_googleapis_com_topic_message_sizes_count[5m]) )      - name: HTTPRules        rules:          - record: job:http_requests:rate5m            expr: sum by (app) (rate(http_requests_total[5m]))
Now you should see the recording rules on Prometheus/Grafana:
In order to calculate z-score, we need to compute the mean and the standard deviation for a specified time period. Since we’re dealing with a web service, using one week’s worth of data is a reasonable time period to compute weekly averages:
NOTE: I’ll just be including PubSub rules for brevity, but the same applies to HTTP requests. After adding the new rules, don’t forget to deploy the new Prometheus server configurations to see changes.
Let’s now compute the weekly average and standard deviation values:
- record: job:pubsub_message_size:rate5m:avg_over_time_1w  expr: avg_over_time(job:pubsub_message_size:rate5m[1w])- record: job:pubsub_message_size:rate5m:stddev_over_time_1w  expr: stddev_over_time(job:pubsub_message_size:rate5m[1w])
Z-score is then our sample data point minus the mean all divided by the standard deviation:
# z-score calculation(job:pubsub_message_size:rate5m - job:pubsub_message_size:rate5m:avg_over_time_1w) / (job:pubsub_message_size:rate5m:stddev_over_time_1w)
Once we graph the z-scores, it’s easy to spot anomalies that fall outside our +/- 3σ range:
One note of caution: using z-scores to determine outliers only works if our underlying assumption is correct — that is our aggregated data is normally distributed. There are multiple ways to test for normal distribution, but you can simply plot the aggregated recording rules to visually determine.
If your data doesn’t seem normally distributed, you can calculate z-scores to see if the range stays within ~-3.5 to +3.5. If you see extremely high or low values, the data is too skewed and may need to resort to data transformations to normalize the dataset.
Finally, we can use this data to create alerts to notify when anomalies are detected. Simply add an alerting rule when the z-score falls outside 3σ:
- name: AnomalyDetection  rules:  - alert: PubSubSizeAnomalyDetected    expr: abs( (job:pubsub_message_size:rate5m - job:pubsub_message_size:rate5m:avg_over_time_1w) / (job:pubsub_message_size:rate5m:stddev_over_time_1w) ) > 3    for: 10m    labels:      severity: warning    annotations:      summary: "Anomaly detected in queue {{ $labels.topic }}"      description: "The queue size saw a huge spike or drop off - check the corresponding logs for that service"
We can complement this alert with simple threshold checks (e.g. alert when the number of undelivered messages is greater than 500) to reduce false positives and only alert when a true anomaly is detected. You can also add in other statistical methods like seasonality to account for traffic levels during the week vs. on the weekend.
On the next and final post of the Practical Monitoring with Prometheus and Grafana series, we will secure the Grafana dashboard with GCP’s IAP service to protect our ingress.
You can find other posts in this series here:
Part I: Installing Prometheus + Grafana via Helm in 5 Minutes
Part II: Using Prometheus blackbox exporter for free uptime checks
Part III: Applying simple statistics for anomaly detection using Prometheus
Part IV: Securing Grafana with Identity-Award Proxy