Monitor Spark (Streaming) with Prometheus
By: Salohy Miarisoa
Tags: Spark, Docker, Spark Streaming, Metrics, Prometheus

Monitor Spark (Streaming) with Prometheus

If you are just starting with spark-streaming and you want first to try it out using docker, then this post is made for you. In this post we will try out spark-streaming with a standalone spark cluster.

Thanks to the docker images from big-data-europe, using a standalone spark cluster becomes easier and it just helps us to be faster, when trying out our spark-streaming application. For this post, we assume, that we understand the spark runtime architecture.

Start standalone spark cluster using docker

Here is how we proceed: We are going to have a master container called spark-master, two workers called spark-worker-1 and spark-worker-2, one application written in java or scala, which we pack in a docker container as well, let’s call it spark-app.

First we will start a compose file with the master and the two workers .

Spark Standalone compose file

Then we launch our spark application by running the third container spark-app. Check out the spark streaming app with its dockerfile here.

spark submit with docker

With that done, we have a running standalone spark streaming application. Now we want to expose the application metrics to prometheus.

Expose spark (streaming) metrics to Prometheus

Spark offers a wide monitoring and instrumenting possibilities. Unfortunately it does not include prometheus. So in oder to be able to store monitoring data into prometheus, we need a workaround. According to the spark documentation, it supports sink to graphite. So we will take advantage of that and a package called prom/graphite-exporter, which is an exporter for graphite metrics to prometheus.

The idea is simple, we allow spark to sink metrics to graphite and we expose those metrics to prometheus using the package. Let’s see first how to expose the metrics to prometheus using graphite-exporter.

Expose metrics to prometheus

Go ahead and add graphite-exporter service to our docker-compose file.

This service will expose list of metrics on localhost:9190/metric ,which can be directly scraped by prometheus as usual. The graphite_mapping.conf file labels the metrics, so that we can fully make use of prometheus query language. Below is an example of its content:

maps metrics from graphite

With that done, our graphite-exporter is ready. So let us now enable spark metrics to sink to Graphite.

Enable spark metrics to sink to Graphite

Doing that with spark submit in docker is straightforward. Just create a file called metrics.properties

spark metrics properties file

When running the application with docker, mount that metrics into $SPARK_HOME/conf it will be automatically read.

spark submit using docker

Having that done, we can start to pull the metrics from prometheus and display them on Grafana. Have a look at my post about prometheus and grafana to set up your monitoring. Check out here how the complete compose file looks like.

And that’s it. In a nutshell, we learned how to lunch a spark-streaming application using docker and expose the monitoring metrics to prometheus. I hope it helps you to get started. You can check the links bellow for more inputs.

See ya 👋